‘Publish or perish’ blamed for extreme results

Methodological controversy and a “publish or perish” culture explain why papers from US behavioural studies are more likely to exaggerate findings.

August 28, 2013

That is the conclusion of research by Daniele Fanelli, a Leverhulme early-career fellow in science, technology and innovation studies and John Ioannidis, C. F. Rehnborg professor in disease prevention at Stanford University.

The researchers looked at nearly 1,200 studies in health-related fields which were combined into 82 meta-analyses published between 2009 and 2012.

Their findings confirm earlier suspicions that studies based on behavioural, as opposed to physiological, data are more inclined to report “extreme effects”.

This is assessed according to the extent to which each paper’s reported results deviated significantly from the size of effect reported in the meta-analysis, which the researchers assume “should approximate the true effect that primary studies were trying to measure”.

The paper, US studies may overestimate effect sizes in softer research, published in Proceedings of the National Academy of Sciences of the United States of America, does not report the same effect in non-behavioural research.

The authors suggest the exaggeration of results is particularly rife in behavioural studies due to the field’s complexity and its lack of consensus over the correct methodological approaches.

“When choices are less rigidly determined by theory, they are more likely to be influenced, consciously or unconsciously, by scientists’ own beliefs, expectations, and wishes, and the most basic scientific desire is that of producing an important research finding,” the paper says.

The authors find that the tendency to exaggeration was particularly prevalent in US-based studies, which were also more likely to deviate in the direction predicted by the paper’s hypothesis. 

They suggest this is explained by “the fact that researchers in the US have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards.

“This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesising, and other more or less questionable practices.”

paul.jump@tsleducation.com

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

Daniel Mitchell illustration (29 June 2017)

Academics who think they can do the work of professional staff better than professional staff themselves are not showing the kind of respect they expect from others

As the pay of BBC on-air talent is revealed, one academic comes clean about his salary

A podium constructed out of wood

There are good reasons why some big names are missing from our roster

Senior academics at Teesside University put at risk of redundancy as summer break gets under way

Thorns and butterflies

Conditions that undermine the notion of scholarly vocation – relentless work, ubiquitous bureaucracy – can cause academics acute distress and spur them to quit, says Ruth Barcan