Citation counts as peer review

July 16, 2015

In the continuing debate on the appropriateness of metrics as a tool in the evaluation of research quality, particularly for the next research excellence framework, citation counts are frequently compared and contrasted with the “gold standard” of peer review (“The weight of numbers”, Features, 9 July). It is considered axiomatic that any deviation between what is termed peer review and citation numbers indicates that metrics are an inadequate tool. What is generally ignored is that article citation counts themselves encompass two levels of peer review. The first is that a cited article will have been reviewed prior to its acceptance by a journal, which often involves several referees who will have been selected for being close to the specifics of the article. The second level of review is that citations themselves indicate that an article has been read and its content considered to be significant by the peers who then cite it.

The “gold standard” peer review associated with the REF means in practice that several hundred articles are read over a period of a few months, and although they will be within the broad subject area of the panel, some, and perhaps many, will be at the periphery of (or beyond) the expertise of the individuals who are assessing them – as those who have sat on panels, whether for the REF or the research councils, are aware. Indeed, in many cases, assessment would be by those sufficiently distant from the specific topic of the article that authors would regard them as inappropriate as referees were they to be asked to review the same material for a journal. It is therefore odd to argue that peer review by panel members is somehow more robust and reliable than citation numbers.

Citations do have their weaknesses for the REF, including being more relevant to some fields than to others, and clearly the most recently published articles will have had too little time to be cited. Perhaps the strongest argument against the widespread adoption of citation numbers for the REF is described by Howard Hotson (“A metrics-based REF: from sexennial pain to permanent headache”, Opinion, 9 July) – that this may well lead to ever more frequent cycles of evaluation.

Paul Trayhurn
Research professor
University of Buckingham

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Retired academics calculating moves while playing bowls

Lincoln Allison, Eric Thomas and Richard Larschan reflect on the ‘next phase’ of the scholarly life