Pre-registration would put science in chains

The pre-registration of study designs must be resisted, says Sophie Scott

July 25, 2013

Science is not well served by people deciding that their methodology is the only legitimate one

As concern grows about questionable practices and outright misconduct in the life sciences, the pre-registration of study designs and hypotheses is being wrongly touted as the panacea.

The campaign’s latest push came in an open letter to The Guardian last month written by Chris Chambers, research fellow at Cardiff University, and Marcus Munafo, professor of biological psychology at the University of Bristol, which was supported by more than 80 signatories.

Drawing on a paper that asked psychologists to self-report their own dubious behaviour, they argue that large numbers of life scientists cherry-pick data, hide null results, fail to employ adequate statistical power and reinvent the aims of studies after they have been completed to make it look as though unexpected findings were predicted.

They claim that pre-registration, which would involve journals accepting future papers based on the design of experiments rather than their results, would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced.

However, there are numerous problems with the idea. Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective. And the commitment to publish with the journal concerned would curtail researchers’ freedom to choose the most appropriate forum for their work after they have considered the results.

With no results to go on, reviewers would be more likely than ever to rely on reputation, which would count against junior scientists. Unsympathetic ones would also be handed the chance to veto studies at the outset. In addition, the requirement to refine studies and their interpretation prior to data collection would prevent us from learning from our mistakes along the way.

Moreover, in my fields (cognitive neuroscience and psychology), a significant proportion of studies would simply be impossible to run on a pre-registration model because many are not designed simply to test hypotheses. Some, for instance, are observational, while many of the participant populations introduce significant sources of complexity and noise; as introductions to psychology often point out, humans are very dirty test tubes.

I am also very uncomfortable with the model’s implication that hypothesis testing is the only correct way of doing science. This may be true for the clinical trials from which pre-registration takes its inspiration, but we have known since Thomas Kuhn that scientists don’t just proceed, study by study, testing individual hypotheses.

In online discussions, Chambers has claimed that pre-registered studies would have “a substantially higher truth value than regular studies” because of measures such as a requirement for authors to declare whether they are discussing results they did not predict. But this sort of language is deeply misleading. Most scientific studies are “wrong” in the long term. Science is a process rather than a method of finding out all the things that are true: a process during which we must be allowed to run studies in which we get things wrong, change our minds and are led in directions we didn’t expect.

If we allow to pass uncontested the claim that the pre-registration model is a gold standard, we will permit the denigration of the vast majority of great research and allow a number of serious constraints to be placed on it.

Science is advanced by communities of researchers and is not well served by people deciding that their methodology is the only legitimate one. Instead, our responsibility is to do the best science we can; to be open-minded and interested in the findings of others; and to do our best to support the careers of those who come to work with us.

That way, there is a fighting chance that we will be remembered for the good stuff that we do. After all, even Newton sometimes employed dubious methodologies. His celebrated physical laws were supported by data, but history tends to overlook his equally enthusiastic pursuit of alchemy, which swam in a sea of null results.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Reader's comments (2)

Pre-registration would force reviewers to rely on reputation of the authors alone to judge a paper's merit? It would seem there would be a need to blind reviewers to author names and vice versa. Isn't this the current system? I also question the premise that if someone's design and theory is approved for a good journal, but they end up with "great" (I'm assuming you mean statistically significant) results, you'd be shackled to publish in the journal that ok'd the design. Do great results offset good designs? Seems like you'd be rewarding sampling error and/or questionable research practices. If the argument is that we should not be so reliant on the hypothetico-deductive model & NHST, then I'm behind you 100%, but I think this article is easily misinterpreted as "everything's fine, no change needed" when in fact the system is broken and post hoc data and hypothesis tinkering is the norm, not the exception.
I think that several of your criticisms of pre-registration may come from misunderstanding of what pre-registration is. Pre-registration does not forbid descriptive or exploratory research; it only makes certain that exploratory research is not labeled as confirmatory research. You mention that populations introduce complexity and noise, that humans are "dirty test tubes." At what point does a scientist know she has parsed out the complexity and noise rather than fit conclusions to noise? Without a confirmatory, preregistered replication, it is sometimes hard to tell what is truth and what is post-hoc rationalization. It is one thing to say "Exploring the data, it appears the effect may hold only for men but not for women," and it is another to say "Consistent with our predictions, the effect held only for men but not for women."

Have your say

Log in or register to post comments

Most Commented

James Fryer illustration (27 July 2017)

It is not Luddism to be cautious about destroying an academic publishing industry that has served us well, says Marilyn Deegan

Jeffrey Beall, associate professor and librarian at the University of Colorado Denver

Creator of controversial predatory journals blacklist says some peers are failing to warn of dangers of disreputable publishers

Hand squeezing stress ball
Working 55 hours per week, the loss of research periods, slashed pensions, increased bureaucracy, tiny budgets and declining standards have finally forced Michael Edwards out
Kayaker and jet skiiers

Nazima Kadir’s social circle reveals a range of alternative careers for would-be scholars, and often with better rewards than academia

hole in ground

‘Drastic action’ required to fix multibillion-pound shortfall in Universities Superannuation Scheme, expert warns