Peer review: democracy, sausages and lotteries

MARK GRIFFITHS

'I'm afraid this paper reminds me of the sausages I had in England 35 years ago... '

I have taken a backseat interest in the peer review system for many years, wondering whether it is the best way to judge quality. I have come to the conclusion that although we all claim to be objective, decision-making is, nevertheless, subjective in the final analysis. For instance, take the example of a literature review on computer game-playing I submitted some while back. The first reviewer said: "I've looked at [the paper] straightaway and I don't like it. It is an uncritical review of the material offering little in the way of 'handles' on the area. Do send it elsewhere as maybe I'm having a blank on it. It just seems so poor." The second reviewer, however, commented that the paper was "an excellent review article with both detailed references to the current research literature and a comprehensive and comprehensible commentary. The theme will be of wide interest to journal readers."

Why do we have peer review if it can produce such wide variations of opinion? Nor is this a one-off instance. Take this example: Reviewer 1: "I cannot recommend accepting this article for publication in its current form. The style of the article is weak and inconsistent. The author's discussion was superficial."

Reviewer 2: "This is an economically written, admirably clear and very helpful paper. There is only one place I would like a little more detail."

Perhaps a more interesting example came from a paper where I had three reviewers who all had different opinions about a collection of case histories of pathological gamblers. It was obvious that two of the reviewers favoured a quantitative approach while the third one was entirely sympathetic to my qualitative approach. The first review was hardly the academic criticism I was expecting: "I am afraid this paper reminds me of the sausages I encountered in England 35 years ago when postwar austerity was still the order of the day, 25 per cent meat and 75 per cent filler." The other two reviewers differed significantly over the number of subjects. One said: "This study was worthwhile but the poor response rate should be clarified further. Nineteen subjects is not enough." The other said: "I believe the strong point of your study is the rich data of personal histories. There is a problem however with the assumptions you make about methodology causing you to justify your small sample size. You feel you can only make tentative conclusions - I disagree. I believe you have all the data you need and can make powerful conclusions if you adopt a qualitative approach. Stop apologising for your small sample size." Thankfully this paper was published but it did highlight the subjective nature of biases about research methodology.

Finally, it is worth mentioning the impact an unfair negative review can have on a person. A paper I submitted during my PhD contained a personal account of problems in participant observation relating to my fruit machine gambling experiences as a "punter". It was never meant to be wholly academic (this I pointed out in the first line). One reviewer said: "Though this is a pleasant and interesting story, it does not belong in this journal. It would be more suitable for a newsletter." The other reviewer was harsher: "I doubt if many professional psychologists are at much risk of becoming pathological gamblers because they work with gamblers, but if they go off gambling themselves - well, that's foolish, just like drinking too much or taking addictive drugs I If he wants to become a professional psychologist - not something I would suggest for him at this time - he should seek professional counselling for emotional problems. It would not harm him to spend more time in the library, laboratory and lecture hall and less time hanging around pubs and gambling halls. Is there an alcohol problem in there somewhere?"

To state that I might have had a drink problem was totally unfounded. If the American reviewer had known about the relationship between gaming machines, pubs and student life in the UK, such an assertion would probably never have been made.

Reading these reviews again makes me smile, but only because all of the papers have subsequently been published. The anecdotal evidence I have highlighted may stimulate more debate on what is a very important issue.

Mark Griffiths is lecturer in psychology, Nottingham Trent University.