Alternative metrics should be used by the European Commission alongside expert judgement and other measures of research quality, according to a new report.
The report cautions against relying too heavily on new ways of measuring research when developing the open science agenda in Europe.
It adds that ensuring that data are collected and processed transparently will be crucial to the process of introducing new research measurement indicators.
Carlos Moedas, the European Commissioner for research, science and innovation, has made opening up science in Europe a key objective of his tenure. By doing so, he hopes to make research more accessible to society, and to make the products of research, such as papers and data, available to other scientists as soon as possible.
As part of this, he commissioned an expert advisory group to look at the strengths and weaknesses of alternative metrics. Altmetrics include measurements of social media activity, for example, and can be used to give a broader picture of research impact compared with more traditional measurements such as citation counts and journal impact factors.
The group, led by James Wilsdon, professor of research policy at the University of Sheffield, came to its conclusions by reviewing the literature and evidence submitted to it about how new metrics could help to advance the work on opening up European science.
It found that some alternative indicators of open science are already available. These include measures of document views and download numbers, co-authorship counts and the data on the number of tweets, likes and shares on social media.
But the report, Next-Generation Metrics: Responsible Metrics and Evaluation for Open Science, cautions that these are “not perfect”. “Many available altmetrics are not yet robust or ready to be used for purposes of research evaluation and need to be monitored carefully before they are applied in such contexts,” the authors say.
“Current indicators need to be improved, and should co-exist, with additional indicators and peer review, in the policy and evaluation process,” they add.
Professor Wilsdon told Times Higher Education that the advisory group was not “cheerleading” for alternative metrics. But he added that many of these indicators try to capture information about the broader audience and impact of research, which is a “good thing”.
The European Commission is interested in gaining a more holistic picture of research because it is under pressure to demonstrate the value of research funding. “There is a quite a lot of scrutiny of and pressure on EU research budgets,” Professor Wilsdon added.
Professor Wilsdon said that the Commission should avoid linking metrics to the allocation of funding because this gives people “incentives to game them”.
Instead, metrics should be used “in intelligent ways to measure what goes on in research once it has been funded”, he continued.
The report adds that an open, transparent and linked data infrastructure will be needed if next-generation metrics are to be used in research evaluation. “How underlying data are collected and processed is crucial,” the group says.
