Wellcome warning about metrics morass

Altmetrics forum weighs the value of impact data from blogs, ‘likes’, web traffic. Paul Jump reports

October 2, 2014

Researchers’ creativity could be undermined by too many new metrical approaches to research evaluation, the director of the Wellcome Trust has warned.

Jeremy Farrar told a conference in London last week that so-called altmetrics - research metrics that go beyond traditional citation counts - offered the possibility of “driving change” by better connecting researchers with each other and with the wider community. But he said that the current burden on the research community was “massive”.

“We are in danger of overburdening it with ever more approaches, and it is on the edge of not being able to cope…such that we will destroy [its] creativity and innovation,” he told the 1:AM conference on altmetrics on 25 September.

Altmetrics often involve counting the number of mentions or “likes” of papers on social media. Some commentators have suggested that this could provide an early indication of how many citations a paper is likely to accrue ultimately. But Euan Adie, founder of altmetrics provider Altmetric, admitted that the correlations were “very weak”.

Altmetrics were better suited to assessing the broader social impact of papers, he said, adding that while analysing social media was useful, the movement had to go further.

He said that his firm was also assessing mentions on blogs and review websites such as Faculty of 1000, as well as policy documents - although he said it would take human intervention to ascertain whether the policy had been implemented.

Andrea Michalek, co-founder and president of Plum Analytics, said that her company was also counting comments and reviews, Wikipedia mentions and bookmarks in reference managers.

Adam Dinsmore, evaluation officer at the Wellcome Trust, said that the trust used altmetrics to flag up potentially interesting “narratives” about the influence of research it funded. However, he said, a high altmetric score did not necessarily imply that the research had made a crucial scientific impact, and it was important to consider whose attention papers had caught. For example, the trust searched for mentions of its papers in academic syllabuses and on websites with .ac and .gov domain names.

Liz Philpots, head of research at the Association of Medical Research Charities, said that she used altmetrics to monitor public discussion of funded research, but warned that assessing researchers on the basis of altmetrics would risk judging them “on the basis of their communication skills, not their research skills”.

James Wilsdon, professor of science and democracy at the University of Sussex, is chairing an independent review of the use of metrics in research assessment for the Higher Education Funding Council for England. It has received 152 written responses, which will soon be posted online. Preliminary conclusions will appear in March, followed by a final report in June.

He said that the “incredible dynamism and innovation” in altmetrics was “genuinely exciting for all of us who want to see a creative, outward facing, socially engaged and impactful scientific enterprise”. But he echoed Professor Farrar’s concern that it also had the potential to overburden researchers with “yet more complex systems and tools for them to worry about”.

paul.jump@tesglobal.com

 

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest