Altered states for the lawmakers of the new measurement frontier

Citations and impact factors are old hat, says Matthew Gamble; the Web 2.0 generation needs metrics to match today’s scholarship

July 28, 2011

 



Credit: Paulo Buchinho


As a young academic, I am reliably informed that the landscape of scholarly communication is not what it was 20 years ago. But, despite all that has changed, it seems that we still largely rely upon the same tired and narrow measures of quality and academic impact - namely, citation counts and journal impact factors.

As someone who has used the internet in almost every aspect of their academic work to date, it’s hard for me to ignore the fact that these mechanisms, in predating the web, largely ignore its effects.

By holding up these measures as incentives, we appear to have our eye firmly fixed on the hammer and not the nail, adjusting our research habits in order to maximise scores and ignoring issues such as why we publish in the first place.

Some serious academic discussion takes place online. (If you are in any doubt, I recommend a visit to the blog of Timothy Gowers, Rouse Ball professor of mathematics at the University of Cambridge and a Fields Medal recipient.) The utility of the web as a platform for scholarly communication should come as no surprise: indeed, when Sir Tim Berners-Lee sat in his office at Cern and created it, he was motivated by a desire to do just that.

Increasingly, many of our scholarly activities are transferring to the web. We write academic blogs; the backchannel of conferences is played out through Twitter; and reference managers such as Mendeley manage a library of close to 100 million papers for more than 1 million academics. All of this activity indicates impact. What is more, because it is on the web, we can observe and measure it.

In assessing impact, we can and should take advantage of these emerging traces of scholarship - but to do so requires us to broaden both what we measure and how we measure it.

The “alt-metrics” community has recently emerged in an effort to achieve this. Complete with a manifesto - at altmetrics.org - this community is striving to understand and measure the products and practices of scholarly communication on the web. It is called “alt” because the practitioners are looking to move beyond the citation-based measures of impact that have dominated the quantitative study of scholarship to date.

Jason Priem, a PhD student at the University of North Carolina at Chapel Hill and the initiator of the alt-metrics movement, talks passionately about a “second scientific revolution” facilitated by the web, and sees alt-metrics as critical to realising it. Key to his vision is the recognition that scholarship is a multidimensional activity, not confined solely to the publication of academic papers. As such, the assessment of scholarship should reflect this and integrate as many metrics as possible.

By harnessing the vast and diverse amount of scholarly activity on the web, alt-metrics promises to provide timely indications of impact and capture practices that have hitherto gone unrewarded.

Readermeter.org illustrates perfectly the immediate possibility of alt-metrics by providing real-time readership counts of literature harvested from online reference-management services. While citation counts can take time to accumulate, this website offers authors an instant impression of their impact.

Perhaps the clearest demonstration of the bold ambition of alt-metrics lies in the work of Samuel Arbesman. At the recent inaugural alt-metrics workshop, Arbesman, a research Fellow affiliated with Harvard University’s Institute for Quantitative Social Science, discussed his efforts to define, quantify and predict scientific discoveries. He claims to have roughly predicted the likely timing of the discovery of the first Earth-like planet, presenting what appears to be - at least in the case of funding decisions - the holy grail of metrics. His ambition serves to highlight just how narrow the approaches of old have been.

Of course, significant challenges will need to be addressed by the alt-metrics community - not least the issue of “gaming” (when academics seek to “play” the system). However, Priem believes that this problem could in part be solved by using an array of diverse metrics.

It will be more difficult still to displace existing measures. The journal impact factor, for all of the criticism, has persisted in part because of its simplicity.

Sceptics have warned that any move away from traditional peer-reviewed journal output will lead to an academic “Wild West”. With alt-metrics, however, the hope is that we as academics can regain control of the mechanisms by which we are assessed - and become the lawmakers of this new frontier.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 10 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Business Development Officer YORK ST JOHN UNIVERSITY
Lecturer in Linguistics YORK ST JOHN UNIVERSITY
Associate Professor/Professor EDINBURGH NAPIER UNIVERSITY

Most Commented

Brexit, EU referendum

Joanna Williams voted Leave, and has been left disappointed by the academy’s reaction to the EU referendum result

Elly Walton illustration (7 July 2016)

Researchers in the liberal arts seem to have made it their mission to communicate in the most obscure fashion, says Zachary Foster

Daniel Mitchell illustration (14 July 2016)

Frank Furedi says the mournful mood on campus and the disparagement and silencing of Leave supporters betray an isolated scholarly class

Michael Parkin illustration (7 July 2016)

Rising immigration-related costs and lack of employer support send an unwelcoming message to international staff, says Jason Danely

Huge eyes peering through door windows

Some scholars claim plans are more about improving staff visibility to students than brightening corridors