Hepi has 'doubt' over citations

Citations don't measure quality, but they could still form part of a revamped research funding system says think-tank. Zoe Corbyn reports.

The quality of academics' research cannot be judged on the basis of the number of times their papers are cited by peers, a leading higher education think-tank has said.

The Higher Education Policy Institute says in a report this week that "there must be real doubt" over the validity of plans by funding chiefs to allocate about £1.4 billion a year in "quality research" funding on the basis of journal article citations, as is planned after next year's research assessment exercise.

Hepi's report on the Higher Education Funding Council for England's proposed research excellence framework (REF) also implies that Hefce may be ignoring the advice of its own "appointed experts" in pursing the change.

"Citation analysis does not measure quality, so there must be real doubt about whether it can be used as a basis for allocating QR (quality research funding)," the report says. "This is awkward in light of the commitment of Hefce to continue to allocate research funds on the basis of quality."

Hepi's director, Bahram Bekhradnia, who as a former Hefce director was the architect of the current RAE, said this conclusion "merely reiterated" what had already been said by a group of academics from the University of Leiden who were commissioned to undertake a scoping study on bibliometrics for Hefce.

The Leiden group said that citation analysis measures research impact, not quality, and that it should be used only to allocate funding in conjunction with peer review.

In the report, Hepi argues that the results of citation analysis and the current RAE, which most agree does measure quality, should be considered side by side to see how closely they match up.

If there is a discrepancy, it says, the new system would need to be modified to one of peer review informed by citations, as proposed by the Leiden study.

"If the match is close, then citations may be a sustainable basis for judging and allocating funds for research, but judgment must be suspended until then," Mr Bekhradnia said.

Responding to the Hepi report, Rama Thirunamachandran, Hefce's director of research, said the proposed REF and the current RAE both made judgments about "research quality".

"The REF is a judgment of quality based on hundreds of peers citing a body of research produced by a unit, (whereas the) RAE is a judgment based on the views of panels of a dozen or so people," he told The Times Higher , adding that a number of studies have shown a "significant positive correlation" between the two approaches.

"We would not expect the two approaches to produce identical results. The important thing for Hefce is to establish an REF that commands the confidence of the higher education sector," he said.

He denied that Hefce was acting at odds with the advice from the University of Leiden.

"The Leiden report recommends that experts with subject knowledge should be involved in interpreting the data. It does not recommend panels re- reading already peer-reviewed articles in peer-reviewed journals," he said.

During the consultation, Hefce is asking for the sector's input into the role that six "expert panels" - each overseeing a large science area - should play in the REF.

It envisages respondents advising on how citations should be combined with other indicators of research income and postgraduate student numbers to determine final funding allocations.

"It is a very narrow role for these panels," Mr Bekhradnia said.

On a positive note, he congratulated Hefce on moving away from its original proposal to use only research income to calculate funding. "It is a great step forward," he said.

zoe.corbyn@thes.co.uk

 

Scotland, Wales to 'wait and see'

The Scottish Funding Council has said it will proceed more cautiously than its English counterpart when it comes to using the research excellence framework to inform funding decisions.

David Gani, the council's director of research policy and strategy, told The Times Higher that the SFC wanted to see a UK-wide system for measuring research quality but, as now, differences would remain in how that would inform bud- get allocation.

His comments came as the Scottish Funding Council launched the consultation exercise on the REF north of the border.

Dr Gani said: "We are committed to developing the research excellence framework and strongly believe that there should be a UK-wide system to measure research quality.

"However, we will wait to see how robust the citation system is before we consider using it to inform our funding decisions. We will proceed more cautiously than the Higher Education Funding Council for England in this respect."

A spokesman for the Higher Education Funding Council for Wales said: "We are also committed to a UK-wide system of assessing research quality, although we would take our own decisions on how to use the outcomes to inform our funding allocations."