‘Publish or perish’ blamed for extreme results

Methodological controversy and a “publish or perish” culture explain why papers from US behavioural studies are more likely to exaggerate findings.

August 28, 2013

That is the conclusion of research by Daniele Fanelli, a Leverhulme early-career fellow in science, technology and innovation studies and John Ioannidis, C. F. Rehnborg professor in disease prevention at Stanford University.

The researchers looked at nearly 1,200 studies in health-related fields which were combined into 82 meta-analyses published between 2009 and 2012.

Their findings confirm earlier suspicions that studies based on behavioural, as opposed to physiological, data are more inclined to report “extreme effects”.

This is assessed according to the extent to which each paper’s reported results deviated significantly from the size of effect reported in the meta-analysis, which the researchers assume “should approximate the true effect that primary studies were trying to measure”.

The paper, US studies may overestimate effect sizes in softer research, published in Proceedings of the National Academy of Sciences of the United States of America, does not report the same effect in non-behavioural research.

The authors suggest the exaggeration of results is particularly rife in behavioural studies due to the field’s complexity and its lack of consensus over the correct methodological approaches.

“When choices are less rigidly determined by theory, they are more likely to be influenced, consciously or unconsciously, by scientists’ own beliefs, expectations, and wishes, and the most basic scientific desire is that of producing an important research finding,” the paper says.

The authors find that the tendency to exaggeration was particularly prevalent in US-based studies, which were also more likely to deviate in the direction predicted by the paper’s hypothesis. 

They suggest this is explained by “the fact that researchers in the US have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards.

“This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesising, and other more or less questionable practices.”

paul.jump@tsleducation.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

United Nations peace keeper

Understanding the unwritten rules of graduate study is vital if you want to get the most from your PhD supervision, say Kevin O'Gorman and Robert MacIntosh

David Parkins Christmas illustration (22 December 2016)

A Dickensian tale, set in today’s university

Eleanor Shakespeare illustration (5 January 2017)

Fixing problems in the academic job market by reducing the number of PhDs would homogenise the sector, argues Tom Cutterham

poi, circus

Kate Riegle van West had to battle to bring her circus life and her academic life together

man with frozen beard, Lake Louise, Canada

Australia also makes gains in list of most attractive English-speaking nations as US slips