‘Publish or perish’ blamed for extreme results

Methodological controversy and a “publish or perish” culture explain why papers from US behavioural studies are more likely to exaggerate findings.

August 28, 2013

That is the conclusion of research by Daniele Fanelli, a Leverhulme early-career fellow in science, technology and innovation studies and John Ioannidis, C. F. Rehnborg professor in disease prevention at Stanford University.

The researchers looked at nearly 1,200 studies in health-related fields which were combined into 82 meta-analyses published between 2009 and 2012.

Their findings confirm earlier suspicions that studies based on behavioural, as opposed to physiological, data are more inclined to report “extreme effects”.

This is assessed according to the extent to which each paper’s reported results deviated significantly from the size of effect reported in the meta-analysis, which the researchers assume “should approximate the true effect that primary studies were trying to measure”.

The paper, US studies may overestimate effect sizes in softer research, published in Proceedings of the National Academy of Sciences of the United States of America, does not report the same effect in non-behavioural research.

The authors suggest the exaggeration of results is particularly rife in behavioural studies due to the field’s complexity and its lack of consensus over the correct methodological approaches.

“When choices are less rigidly determined by theory, they are more likely to be influenced, consciously or unconsciously, by scientists’ own beliefs, expectations, and wishes, and the most basic scientific desire is that of producing an important research finding,” the paper says.

The authors find that the tendency to exaggeration was particularly prevalent in US-based studies, which were also more likely to deviate in the direction predicted by the paper’s hypothesis. 

They suggest this is explained by “the fact that researchers in the US have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards.

“This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesising, and other more or less questionable practices.”

paul.jump@tsleducation.com

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Assistant Recruitment - Human Resources Office

University Of Nottingham Ningbo China

Outreach Officer

Gsm London

Professorship in Geomatics

Norwegian University Of Science & Technology -ntnu

Professor of European History

Newcastle University

Head of Department

University Of Chichester
See all jobs

Most Commented

men in office with feet on desk. Vintage

Three-quarters of respondents are dissatisfied with the people running their institutions

students use laptops

Researchers say students who use computers score half a grade lower than those who write notes

Canal houses, Amsterdam, Netherlands

All three of England’s for-profit universities owned in Netherlands

Mitch Blunt illustration (23 March 2017)

Without more conservative perspectives in the academy, lawmakers will increasingly ignore and potentially defund social science, says Musa al-Gharbi

sitting by statue

Institutions told they have a ‘culture of excluding postgraduates’ in wake of damning study