Papers in economics ‘not reproducible’

Fears that discipline is particularly susceptible to statistical ‘hacking’ of data to gain a positive result

October 21, 2015
Trader reacts to DAX index board, Frankfurt stock exchange
Source: Reuters
Misleading models? Results of only 49 per cent of papers could be replicated

At least half of papers in economics are not reproducible, a new analysis has found, suggesting that the “reproducibility crisis” in academia is not confined to lab sciences.

Researchers from the United States Federal Reserve and the Department of the Treasury tried to replicate the results from 67 papers across 13 prestigious journals, but even after contacting authors when necessary, they were successful in only 49 per cent of cases where the data were not confidential and the researchers had the right software to analyse it.

“We assert that economics research is usually not replicable,” the paper concludes.

The findings feed into broader concerns that academics are engaging in statistical sleights of hand, not being open with data and failing to control their own biases in order to get career-boosting positive results in top journals.

Last week, two studies reported that scientists doing experiments with animals were often failing to use well-known techniques – for example blinding themselves to which animals were receiving what drug – to mitigate their biases, and therefore potentially exaggerating the impact of new treatments.

But this new study highlights that the problem of reproducibility might not be confined to the lab.

According to the paper, the main reason for being unable to replicate findings was an inability to find the right data or the computer code that produced the original results, even after contacting the authors. Code was missing crucial functions, or certain variables were absent from the data, the paper says.

However, in nine cases for which the authors of the paper had the right dataset and code, they nonetheless got a different result or the code failed to finish executing, according to “Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not’”.

Sarah Necker, a research associate at the Walter Eucken Institute in Freiburg whose previous research has revealed suspect research practices among economists, said that it was unclear whether the lack of reproducibility was down to the deliberate “hacking” of statistics to get a positive result or to honest errors.

But economics might be particularly open to statistical manipulation compared with other disciplines, Dr Necker explained. For example, an economist could scrutinise data to uncover what causes economic growth and find that it correlates to a particular variable, such as the level of innovation in a society.

This correlation might disappear if a different variable is controlled for, but given that the researcher decides what to control, this affords a great deal of scope for manipulating results, she argued.

Last year, a survey of economists by Dr Necker revealed that more than a third admitted to “searching for control variables until you get the desired results”, even though 85 per cent considered this practice to be unjustifiable.

She added: “Economic methods have become pretty complicated. The datasets are huge. In economics it has become popular to have some fancy [mathematical] method, but that makes it more difficult to replicate.”

Some journals now ask for data to be made available alongside articles, Dr Necker explained, and an online “Replication Network” encourages the practice among economists.

But the incentives for economists to provide open data so that their results can be replicated were still “low”, she said. It was “time-consuming” to provide clean data for public scrutiny, she said, and it increased the risk that errors would be discovered. 

david.matthews@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

POSTSCRIPT:

Print headline: Voodoo economics: most papers in the field are ‘not reproducible’

Have your say

Log in or register to post comments

Featured Jobs

PhD Position in Archaeology and Cultural History

Norwegian University Of Science & Technology -ntnu

PhD position in Energy and Process Engineering

Norwegian University Of Science & Technology -ntnu

PhD position in Energy and Process Engineering

Norwegian University Of Science & Technology -ntnu

PhD position in Industrial Energy Efficiency

Norwegian University Of Science & Technology -ntnu

Postdoc in Traffic Engineering

Norwegian University Of Science & Technology -ntnu
See all jobs

Most Commented

Doctoral study can seem like a 24-7 endeavour, but don't ignore these other opportunities, advise Robert MacIntosh and Kevin O'Gorman

Matthew Brazier illustration (9 February 2017)

How do you defeat Nazis and liars? Focus on the people in earshot, says eminent Holocaust scholar Deborah Lipstadt

Improvement, performance, rankings, success

Phil Baty sets out why the World University Rankings are here to stay – and why that's a good thing

Warwick vice-chancellor Stuart Croft on why his university reluctantly joined the ‘flawed’ teaching excellence framework

people dressed in game of thrones costume

Old Germanic languages are back in vogue, but what value are they to a modern-day graduate? Alice Durrans writes