Altered states for the lawmakers of the new measurement frontier

Citations and impact factors are old hat, says Matthew Gamble; the Web 2.0 generation needs metrics to match today’s scholarship

July 28, 2011

 



Credit: Paulo Buchinho


As a young academic, I am reliably informed that the landscape of scholarly communication is not what it was 20 years ago. But, despite all that has changed, it seems that we still largely rely upon the same tired and narrow measures of quality and academic impact - namely, citation counts and journal impact factors.

As someone who has used the internet in almost every aspect of their academic work to date, it’s hard for me to ignore the fact that these mechanisms, in predating the web, largely ignore its effects.

By holding up these measures as incentives, we appear to have our eye firmly fixed on the hammer and not the nail, adjusting our research habits in order to maximise scores and ignoring issues such as why we publish in the first place.

Some serious academic discussion takes place online. (If you are in any doubt, I recommend a visit to the blog of Timothy Gowers, Rouse Ball professor of mathematics at the University of Cambridge and a Fields Medal recipient.) The utility of the web as a platform for scholarly communication should come as no surprise: indeed, when Sir Tim Berners-Lee sat in his office at Cern and created it, he was motivated by a desire to do just that.

Increasingly, many of our scholarly activities are transferring to the web. We write academic blogs; the backchannel of conferences is played out through Twitter; and reference managers such as Mendeley manage a library of close to 100 million papers for more than 1 million academics. All of this activity indicates impact. What is more, because it is on the web, we can observe and measure it.

In assessing impact, we can and should take advantage of these emerging traces of scholarship - but to do so requires us to broaden both what we measure and how we measure it.

The “alt-metrics” community has recently emerged in an effort to achieve this. Complete with a manifesto - at altmetrics.org - this community is striving to understand and measure the products and practices of scholarly communication on the web. It is called “alt” because the practitioners are looking to move beyond the citation-based measures of impact that have dominated the quantitative study of scholarship to date.

Jason Priem, a PhD student at the University of North Carolina at Chapel Hill and the initiator of the alt-metrics movement, talks passionately about a “second scientific revolution” facilitated by the web, and sees alt-metrics as critical to realising it. Key to his vision is the recognition that scholarship is a multidimensional activity, not confined solely to the publication of academic papers. As such, the assessment of scholarship should reflect this and integrate as many metrics as possible.

By harnessing the vast and diverse amount of scholarly activity on the web, alt-metrics promises to provide timely indications of impact and capture practices that have hitherto gone unrewarded.

Readermeter.org illustrates perfectly the immediate possibility of alt-metrics by providing real-time readership counts of literature harvested from online reference-management services. While citation counts can take time to accumulate, this website offers authors an instant impression of their impact.

Perhaps the clearest demonstration of the bold ambition of alt-metrics lies in the work of Samuel Arbesman. At the recent inaugural alt-metrics workshop, Arbesman, a research Fellow affiliated with Harvard University’s Institute for Quantitative Social Science, discussed his efforts to define, quantify and predict scientific discoveries. He claims to have roughly predicted the likely timing of the discovery of the first Earth-like planet, presenting what appears to be - at least in the case of funding decisions - the holy grail of metrics. His ambition serves to highlight just how narrow the approaches of old have been.

Of course, significant challenges will need to be addressed by the alt-metrics community - not least the issue of “gaming” (when academics seek to “play” the system). However, Priem believes that this problem could in part be solved by using an array of diverse metrics.

It will be more difficult still to displace existing measures. The journal impact factor, for all of the criticism, has persisted in part because of its simplicity.

Sceptics have warned that any move away from traditional peer-reviewed journal output will lead to an academic “Wild West”. With alt-metrics, however, the hope is that we as academics can regain control of the mechanisms by which we are assessed - and become the lawmakers of this new frontier.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Senior Lecturer in Human Genetics LOUGHBOROUGH UNIVERSITY
Senior Lecturer in Biochemistry LOUGHBOROUGH UNIVERSITY
Lecturer in Biochemistry LOUGHBOROUGH UNIVERSITY
Professor in Marketing UNIVERSITY OF GLASGOW

Most Commented

Social media icons

Gabriel Egan laments the narcissistic craving for others’ approval brought on, he says, by the use of social networking websites

Elly Walton illustration (25 August 2016)

Treating students as consumers has precipitated a rush to the bottom to give them exactly what they want, says John Warren

Superhero costumes hanging on a washing line

Senior management do not recognise support staff’s pivotal role in achieving positive student outcomes, administrators say

Man photocopying a book

Students think it ‘unfair’ to be punished for unintentional plagiarism

to write students’ assessed essays in return for cash

Vic Boyd was on the lookout for academic writing opportunities. What she found was somewhat less appetising...