A study confirms that studies into alternative medicine get short shrift from peer reviewers. But, says Geoff Watts, that's not the full story.
Practitioners of acupuncture, homoeopathy, reflexology and other items in the grab-bag of techniques that make up complementary medicine often claim that orthodox, peer-reviewed medical journals will not give them a look in. Even if we carry out rigorous well-planned research, they complain, editors will not touch it.
At first reading, a study published in the April issue of the Journal of the Royal Society of Medicine seems to support that allegation. "A randomised controlled study of reviewer bias against an unconventional therapy" is the title of a report by three doctors who covertly assessed the prejudices of the people who judge the worth and quality of research papers submitted to academic journals.
Edzard Ernst of the department of complementary medicine at the University of Exeter, along with a German colleague, Karl Resch, enlisted the help of John Garrow, an expert on nutrition and obesity. They devised two versions of a short, methodologically sound but entirely fictitious report on a randomised, placebo-controlled trial of an appetite suppressant. The two versions were identical except for the featured remedies: one was an "orthodox" drug, hydroxycitrate; the other was homeopathic sulphur.
To find out how reviewers would respond to these two papers, the group compiled a lengthy list of suitably qualified academics. Half were sent one version of the paper, and half the other. As editor of the European Journal of Clinical Nutrition, Professor Garrow was able to use his own publication's headed note paper and evaluation sheets in soliciting opinions. For the study to be valid, reviewers had to believe that the papers were genuine. With a degree of ingenuity bordering on low cunning, the researchers invented a Ludwig Boltzmann Institute for Metabolic and Nutritional Diseases, which they located in Vienna. There are numerous Boltzmann institutes in Austria, "so a reviewer would be highly unlikely to know for sure that such an institution did not exist". All that remained was to fabricate some suitable authors - which they did by borrowing the names of real people associated with research in Vienna and then changing their initials.
In spite of these elaborate precautions, one of Professor Garrow's regular reviewers did become suspicious. "He phoned up to say that a couple of his colleagues had received copies of what appeared to be the same paper but with slight differences. He wanted to know what was going on." Otherwise, though, the deception went undetected.
Although none of the reviewers complained, the editor of the journal that published the findings is less than happy about the enterprise. In a leader written to accompany the report, Robin Fox comments: "The relationship between editor and reviewer, like that between doctor and patient, depends on trust, and we should be wary of any activity, however well meant, that might vitiate it."
The three authors are aware of the ethical question marks hovering above their study, and they say they discussed them at length. Professor Ernst describes Dr Fox's criticism as "fair comment". But Professor Garrow, who thinks comparisons with the doctor-patient relationship are a shade over the top, goes on to ask how reliable insights into peer review are ever to be gleaned if not with a certain amount of benign deception.
Ethics aside, the results of the study seem to support the claims of the unorthodox practitioners. When reviewers rated the importance of the papers on a scale running from "trivial even if true" to "major contribution to knowledge in the field", the unorthodox study fared significantly worse. In addition to this overall opinion, each reviewer was asked to score various individual elements of the paper, including its methodology, discussion, reliability and so on. In none of these eight separate evaluations did the two versions receive a significantly different score. It was as if, having failed to find specific grounds on which to pick holes in what they had read, reviewers with the version describing the unconventional technique had simply followed a gut dislike. Hence its lower score for overall importance.
"These results," the reviewers say, "suggest that, in this particular setting, a bias against publication of unconventional therapies exists." So, there you have it: QED.
Or do you? Professor Ernst is philosophical: "Reviewing is a judging process, always subjective to a degree. There probably won't ever be a perfect solution." But Professor Garrow argues that the results are not necessarily so damning as they first appear. "It surprised me that the difference between the overall ratings given to the two versions was so small."
If these results are generalised, he says, they would not explain why so little research on complementary medicine is published. "If a complementary practitioner with an unconventional treatment does a trial that is methodologically good and sends it to an orthodox journal, our findings suggest that the chances of its being accepted are good." Bias there may be - but not so extreme as to exclude the unorthodox.
So why the paucity of such studies in academic journals? "Presumably the reason that so few are published," Professor Garrow says, "is that very few are done." The ball, in his view, is back in the complementary half of the court. If unconventional treatments really can be shown to work, a forum is there for publishing the evidence.