Is the journal ratings game good for researchers?

The Association of Business Schools has released its latest ranking of periodicals, but not everyone welcomes it

February 26, 2015

Source: Rex

Read all about it: some argue that the ABS journal guide is ‘intellectually vacuous and empirically false’, while others defend it as a ‘robust’ tool to be used ‘holistically’

With the dust from the research excellence framework having barely settled, the march towards greater measurement and ranking in higher education is clear for all to see.

On this path, one of the most hotly debated innovations has been the emergence of journal lists, which assign ratings to whole periodicals and not just to individual articles.

There are two goals. One is to help researchers to make informed decisions about where they should seek to have their work published. Which journals have the highest standards, largest readerships and most frequently cited papers?

The other is to aid heads of department and other leaders in their attempts to assess the performance of academics, particularly in broad disciplines with numerous sub-specialisms.

For critics, this second mission has become the primary function of journal lists, making them the key consideration in decisions over hiring, promotion and salary review, and leading to researchers being judged largely on where they publish rather than what they publish.

At the front of this debate has been business and management studies. The Association of Business Schools’ Academic Journal Guide 2015, published on 25 February, assesses the quality of 1,401 publications worldwide, drawing on citation scores and the judgements of leading researchers, among other criteria.

It gives the coveted 4* “journal of distinction” title to 33 publications, while 85 are rated 4 – meaning that they contain the “most original and best executed research”.

Some 312 are rated 3, characterised as periodicals of “original and well executed research”, with 481 scoring 2 for “original research of an acceptable standard” and 490 in the bottom tier of 1, with work of a “recognised, but more modest standard”.

The number of journals ranked between 4* and 3 has increased significantly since the last edition of the guide was published five years ago, from 324 to 430. But since the 2010 version rated only 823 titles, this represents a marked fall as a proportion of the total, from 39.4 per cent to 30.7 per cent.

Angus Laing, chair of the ABS and dean of business and economics at Loughborough University, argued that the results were heartening.

“What we are very encouraged by is the number of journals that have been rated in the top bandings; that reflects well on the richness, depth and breadth of the business school community,” he said.

“What I think is important is that there is increasing evidence that UK and [continental] Europe-originated journals are increasingly sitting up alongside American journals.”

The guide will prove particularly useful for early career researchers considering where to seek publication, and for more experienced academics stepping across disciplinary divides, Professor Laing said.

But for Dennis Tourish, professor of leadership and organisation studies at Royal Holloway, University of London, the guide is “intellectually vacuous and empirically false”.

There is a “far from perfect” correlation between the rating of a journal and the quality of its articles according to exercises such as the REF or the 2008 research assessment exercise, he said. And that is after some schools used the guide to decide which outputs to submit.

In the REF, roughly half the business and management outputs considered received identical scores to the 2010 ABS grade of the journal in which they appeared.

Professor Tourish argued that, since researchers are now put under pressure to publish only in 3- or 4-rated journals, the guide acts as an essentially conservative force.

The guide has traditionally been dominated by American titles, which typically emphasise a functional view of the discipline and have declined to explore the impact of the financial crash, he said.

Newer journals find it harder to attract good papers and establish themselves as a result of the guide, according to Professor Tourish.

The guide’s editors are clear that any title not in the 2010 list is limited to a maximum rating of 3 in the latest edition, “other than in exceptional circumstances”.

“People are being shoehorned into publication practices which are not consistent with their values and interests,” said Professor Tourish.

“They have to conform to the publication practices and topics that are considered relevant by elite journals. It limits the scope of academic enquiry in a way I think is harmful.”

This view is echoed by John Mingers, professor of operational research and systems at Kent Business School. He recognises the value of the guide, particularly when looking across a broad discipline, but is concerned about how “hegemonistic” it has become.

“Young researchers who might come along with bright new ideas get put off from being able to publish them,” Professor Mingers said. “I think it has an impact on how much new and groundbreaking research can be done, if we are following fixed pathways.”

For Geoffrey Wood, co-editor of the 2015 edition and professor of international business at Warwick Business School, these complaints are about how the guide is used, not the guide itself.

Good research can be found in all types of journal but does tend to be clustered in certain titles with high standards of peer review, he said.

Identifying these clusters is important, Professor Wood argued, in order to allow academics to plan their careers with confidence and to encourage lecturers to push themselves harder.

In the digital age, the guide acts as a form of Kitemark of quality, protecting young researchers from online journals that may seek to exploit them financially or intellectually, he added.

What is perhaps most important for Professor Wood, however, is the methodology of the guide, which involves consultation with academics and subject associations across a wide range of disciplines.

Prior to the emergence of the ABS ranking, first published in 2007, lists produced by individual business schools held more sway, but these often reflected the interests of the school in question, or even where its leaders published, he said.

“If this is going to happen, it’s better that it happens in a civilised manner, rather than in a barbarous fashion,” Professor Wood said.

Professor Wood agreed with Professor Laing that the information in the guide should not be considered in isolation.

“This is one piece of information – a very robust piece of information – that can be used holistically in making a judgement, either when choosing where to publish or when assessing how an academic’s career is progressing,” Professor Laing said.

“It is certainly not planned as something that should be driving decisions narrowly and in a mechanistic manner. Any dean who does that is a fool.”

View the full list of ABS journal rankings

Times Higher Education free 30-day trial

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Reader's comments (2)

The ABS is the Association of Business Schools, (not Business Studies). It is an association of Deans. The scholarly society for academics is the British Academy of Management. This is their statement: The British Academy of Management, the scholarly association for business and management lecturers and researchers, expressed reservations about the so called ‘ABS List’ of elite journals released today . British Academy of Management Chair, Professor Nic Beech (Dundee University) said : "British business and management research is lauded around the world for its intellectual innovation and pluralism, drawing on the whole range of social science and humanities traditions. It applies these to develop new academic knowledge, and have to have a positive impact on businesses, organisations, managers, and indeed the everyday life of all our citizens. " British Academy of Management President, Professor Sir Cary Cooper (Lancaster University) added: "We are particularly concerned about any ranking list which devalues research with real world impact and insights, in favour of that aimed at topping league tables. Research in business and management must aspire to positive impacts on academic knowledge and on the real world. Ranking lists that favour one way of thinking – for example traditional and narrow ‘functionalist’ approaches – underplay the much broader strengths of a pluralistic approach to business and management research. This kind of innovative scholarship, ironically, is what British business schools – and our members - have a particular worldwide reputation for being good at." BAM Vice-Chair, Research and Publications, Professor Bill Cooke (York University) also stated: ‘lists like this even if well intentioned have problematic unintended consequences. Once the ABS has got over its longstanding technical problems and the list is widely available, we intend to survey our members regarding the methodology of the list, the extent to which it is inclusive or exclusive, and what to do about the list culture’" The Association of Business Schools is an association comprising UK Business School Deans. The British Academy of Management has just under 2000 members, runs two scholarly journals, an annual conference in the UK with over 900 participants and approximately 60 courses and academic events per annum. It has a directly elected council, executive, and President
The latest ABS list is extraordinary first for the bizarre way that it has been released, with a quite blatant data harvesting put in the way of gaining access. But second because it has now become quite clear that it does not stand as a measure of quality as defined by REF judgements i.e. there was a massive divergence between REF panel scores and ABS journal scores. Institutions that slavishly followed the ABS list came a cropper in REF. It is hard to see what useful purpose the list serves except to entrench prejudice. Certainly it bears no relation to substantive intellectual quality. As the comment above shows, it has even been disowned by BAM. It's basically a worthless exercise.