Research intelligence - Opening night, curtain call?

Shakespeare journal screws courage to sticking place on open peer review, but doubts remain. Paul Jump writes

May 5, 2011



Credit: Moviestore Collection/Alamy
No more skulduggery: An experiment in open peer review made some academics feel uneasy about commenting publicly


If something really is rotten in the state of peer review, the latest experiment by the Shakespeare Quarterly suggests that it may be some time before academics embrace a possible remedy.

The Commons Science and Technology Committee recently launched an inquiry into peer review and was scheduled to have its first hearing yesterday. It comes amid concerns - voiced in many of the 93 written submissions the inquiry has received - about the current system's tendency to favour conservative research and the dangers of it being distorted by nepotism, exploited by unethical reviewers or undermined by a lack of appropriate expertise.

Among the inquiry's terms of reference are "the impact of IT and greater use of online resources". Many believe that online open peer review could improve the effectiveness of the process by exposing it to a wider range of publicly identified reviewers.

Last year, the Shakespeare Quarterly became the first humanities journal to experiment with open peer review when it invited comments on seven papers it was considering for inclusion in a special edition on Shakespeare and new media.

David Schalkwyk, editor of the journal and director of research at the Folger Shakespeare Library in Washington DC, described as "terrible" the imbalance of power in traditional "single-blind" peer review, in which the identity of the author is known to the reviewer but not vice versa.

He has adopted double-blind reviewing at the Quarterly (where neither party is known to the other), whereas the open peer-review experiment required both authors' and reviewers' names to be revealed.

Dr Schalkwyk said the experiment had been prompted primarily by a desire to harness the web's potential to support greater scholarly discussion. It also reflected a feeling that "if we were going to talk about Shakespeare and new media, we should practise new media ways of doing things".

However, he was so "pleased and excited" by what transpired - 41 people made more than 350 comments, many provoking responses from the authors - that he decided to repeat the experiment for this year's special edition on Shakespeare and performance.

"The special editions are the best format for open review because there is a particular focus. You can target groups of experts to be your committed reviewers, and you can assume there will be a general body that will also be interested," he said.

Seven articles were posted - although Dr Schalkwyk admitted that several authors had opted for traditional review instead. The final decisions on acceptance will be taken after the authors have had the chance to revise their essays in light of the comments.

Despite "buttonholing" around 25 committed reviewers, Dr Schalkwyk admitted that further cajoling - and a seven-day extension to the original deadline for comments - had been necessary to elicit the final total of 235 comments from 30 people (including the authors).

"We were really worried at one point, but we felt we got good enough commentary by the end," he said, adding that the quality did not differ greatly from that returned by traditional reviewers.

Heart on sleeve proves unpopular

Part of the difficulty of attracting open reviews, he said, related to pressures on academics' time. But he also thought that the "more traditional" subject matter in the second experiment had attracted a different "constituency" of readers who, unlike the new media enthusiasts, did not see "open review as the most normal thing in the world".

He said a lot of academics were reluctant to participate in open peer review because it put "a bit of pressure" on them to "take public responsibility for what they say".

"Our readers aren't untrustworthy or irresponsible, but, structurally, it changes the relationship with the author: it engenders a discussion and many people don't want to get into that," he said.

Dr Schalkwyk is considering allowing online reviewers to be anonymous in a future experiment to see whether it boosts the number of comments.

But decisions about whether a particular special edition should adopt open peer review will now be taken on the basis of whether the theme seems likely to appeal to a web-friendly constituency.

"We are back-pedalling a little," he confessed. He also conceded that open peer review might not turn out to be the best way to promote greater scholarly discussion.

"It may be that it is one thing to create a forum for open discussion and another to ensure work is properly peer reviewed," he said.

The future of open peer review will also be heavily dependent on the survival or otherwise of paper journals, he added. "If everything goes electronic then traditional modes of doing things will seem more old-fashioned. But the fact that we continue to be the lonely scout (in open peer review) is an indication that it isn't catching on."

paul.jump@tsleducation.com

http://tinyurl.com/3lgrocb

http://tinyurl.com/6dxzsls

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

James Fryer illustration (27 July 2017)

It is not Luddism to be cautious about destroying an academic publishing industry that has served us well, says Marilyn Deegan

Jeffrey Beall, associate professor and librarian at the University of Colorado Denver

Creator of controversial predatory journals blacklist says some peers are failing to warn of dangers of disreputable publishers

Hand squeezing stress ball
Working 55 hours per week, the loss of research periods, slashed pensions, increased bureaucracy, tiny budgets and declining standards have finally forced Michael Edwards out
Kayaker and jet skiiers

Nazima Kadir’s social circle reveals a range of alternative careers for would-be scholars, and often with better rewards than academia

hole in ground

‘Drastic action’ required to fix multibillion-pound shortfall in Universities Superannuation Scheme, expert warns