With the rules for the 2008 RAE in place, Anthea Lipsett and Tony Tysome assess the key changes and their impact on departments and individuals
The final ground rules that will be used to judge research by UK academics in the 2008 research assessment exercise were unveiled this week. They will provide guidance for university submissions that will be made across 67 subject areas by October 31, 2007.
Universities will pore over the criteria in the coming weeks to examine the final decisions - taken after consultation with the academic community - on critical issues such as the treatment of "super-books", new academics and applied research. Some of the rules apply across all disciplines, while others will vary between the 15 main panels of academic referees and the 67 sub-panels overseeing the assessment.
One key change to the draft criteria is that early-career researchers will have to submit two pieces of research work for evaluation rather than the four expected of established academics. Judie Newman, chair of Panel M for languages, said the previous expectation of four top-quality examples of work was a "crushing" requirement for new academics. "(The change) makes it a much more friendly exercise for early-career people. They'll be able to develop professionally without feeling pressure."
Academics who have missed work because of maternity leave or prolonged illness will not have to submit four research papers.
On the issue of super-books, panels will not be prescriptive. Such works will be judged individually to see if they merit being counted as more than one submitted academic paper.
Referees will be told to assess research on its quality and not on the journal in which it appeared. "We are asking them as an expert panel to make judgments on research rather than on journals with a high impact factor," said Ed Hughes, the RAE manager.
But perhaps the most crucial decision of all - which academics qualify for the new RAE ratings (ranging from 1* to 4*) - will be left to the discretion of individual panels. Each has detailed what a profile rating represents in terms of academic standards.
The full criteria can be found at www.rae.ac.uk
Social sciences: no appetite for games
With the rules for the 2008 RAE in place, Anthea Lipsett and Tony Tysome assess the key changes and their impact on departments and individuals
RAE panel members in the social sciences have warned they will take a dim view of institutions and departments that indulge in "game-playing" to try to boost their ratings.
David Otley, chair of Panel I for economics and econometrics, accounting and finance, business and management studies, and library and information management, said he would "strongly suggest" that institutions entered all their research-active staff to add to their expected RAE-related income and their "environment" standing.
He said: "People should not go in for game-playing. There are lots of vice-chancellors who seem to be using the RAE as a political tool and an excuse to shed staff who are not 3* or 4*. There is no rational reason for them to do so."
The sub-panels in Professor Otley's area had to strike a compromise on the weightings that are given to research output, environment and esteem - the three measures of research quality used by all panels. They opted for a 70/20/10 spread, placing more emphasis on esteem than the panels overseeing other disciplines.
Professor Otley said: "We are looking for departmental as well as individual esteem. Some panels have suggested that there should be two measures for every person entered, but we feel it isn't really esteem if everyone has it."
Panel J for law, politics and international studies, social work and social policy and administration, sociology, anthropology and development studies, took the view that esteem - as highlighted by, for example, prizes won by academics - was worth only 5 per cent of the total rating of groups of researchers submitted.
Peter Taylor-Gooby, who chaired the sub-panel for social work and social policy, explained: "We think it is quite difficult to measure and (the score) can easily be skewed by just one or two prestigious people."
Professor Otley suggested that some departments might be torn over which publications to submit when it comes to weighing the pros and cons of a super-book.
He said: "In the case of a super-book, we are not talking about quality, but rather the amount of work that has gone into it. We reserve the right to give a super-book only a 1* rating."
Not every social sciences panel expects to have to grapple with the super-book question. Professor Taylor-Gooby said it was not considered to be a significant issue for social work and social policy.
"We have put in a discretionary clause to allow us to consider a bigger piece of work in place of a larger number of publications, so that people have the opportunity to make a case for a super-book.
He said probably the biggest issue for his panel would be wading through the number of publications received.
"We have committed to reading virtually all material submitted, as we have the kind of subjects where you cannot just say it is in journal X so it is world leading."
Arts and humanities: Words count, but quality too
With the rules for the 2008 RAE in place, Anthea Lipsett and Tony Tysome assess the key changes and their impact on departments and individuals
Research assessment panels in the arts and humanities will judge the value of "super-books" case by case and will not allow academics to propose how much they are worth in terms of research outputs.
Martin Daunton, chair of Panel N, covering classics, history and philosophy, said: "It was thought undesirable to have people say what their research was worth. The new scheme is better: you put in four outputs, the sub-panel reads and rates them, then looks again and decides how much weight they should have. So one item might be given 2 or 3 in exceptional circumstances," he said. "It's not the amount of pages, it's the importance of the work in terms of the amount of input and output."
This decision removes the danger of an academic missing out one of their research outputs if the panel disagrees with them that their one large piece of work is worth two outputs for example. It does, however, allow for research profile ratings to be boosted by extra scores for good, large pieces of research. For example, ten people in a department with four outputs each could get a score of 60 rather than 40 if they had submitted several super-books. "It boosts the upper end of the distribution," Professor Daunton said.
According to Judie Newman, chair of Panel L (area studies): "It will be weighed and judged on its quality and intellectual merit. That for us is what the exercise is about: peer review. It's about intellectual scale and scope, not just something to prop your back door open with."
Meanwhile Panel H, covering geography, architecture, archaeology, has tried to make the definitions of ratings as clear as possible. For instance, 4* research will be "internationally outstanding", "innovative" and "potentially agenda-setting", while 3* research will "significantly advance research or policy agendas".
The chair of the panel, Nigel Thrift, said his panel had done all it could to allay any fears that applied and practice-based research will be valued as highly as basic research. "I'm confident that there will be fairness across all styles of research," he said.
Bruce Brown is chair of Panel O, covering art, drama, music. His panel clarified the quality categories and listed which outputs would count.
"Everything will be accepted so long as it's research and the material is there to allow the panel to do its work." Scholarship of research and infrastructure projects such as work on databases that allow for research can also be included.
Professor Newman said the changes to the area studies panel's final criteria were slight. The weightings given to the elements that would make up the research profile rating - research output, environment and esteem - changed from 70, 15 and 15 per cent to 75, 15 and 10 per cent respectively, to allay concerns that the outputs weighting was too low, she said.
Engineering and mathematics: Excellence ÷ time
With the rules for the 2008 RAE in place, Anthea Lipsett and Tony Tysome assess the key changes and their impact on departments and individuals
New academics in mathematics, statistics and computer studies will be expected to produce one academic paper or research output every 21 months they have been in a full-time post.
This is the particular ruling for the 2008 research assessment exercise introduced in response to general concerns in the academic community that early-career researchers would be unfairly penalised in the RAE.
"We responded to the sector's concerns about the need to clarify treatment of (such researchers)," said Bernard Silverman, chair of Sub-panel 22 (statistics and operational research).
The approach is also being adopted by academic referees on Panel F, which covers mathematics, statistics and computer studies.
Elsewhere, panel chairs have said that there were few significant changes to the draft rules that were published last year.
This is the case for Panel G (engineering) and for the sub-panels that sit under it. The scholarly community and the panel referees were satisfied with the initial criteria, said Adrian Long, chair of Panel 26 (chemical engineering).
He said: "We looked at (quality research ratings) very carefully, and in engineering we felt it was best not to define things too precisely. We have endeavoured to get the best possible panel representatives to carry out the peer review, and judgment and flexibility is in everybody's best interest."
The issue of quality research ratings would be considered again at the time of assessment, he said.
Meanwhile, Julia Higgins, chair of Panel E (chemistry, physics and earth sciences), said: "There was a slight compromise between the scientists and the other academics in the panel. We had really very few changes that were made, mostly slight changes in wording, and no great arguments."
Panels also made it clear that all research would be treated equally, which had been a concern in the past.
Professor Silverman said: "Applied research will be treated in every sense as of equal status to theoretical research, and we were delighted to do that. It's not a change in policy, but we needed to be clear on that."
All the assessment panels have added to the definitions of the new profile research ratings after international panel members agreed they stood up to scrutiny. "It was useful to be prompted to clarify that," he said. "We hope these are realistic and rigorous as they apply to our own fields of study."
Panel G (engineering) has given extra detail on how the research esteem and environment elements of the quality profile will be derived.
Half the quality profile for research environment will come from quantitative data on research doctorates awarded per submitted full time staff, with the rest of the profile being derived from other submitted evidence on the research environment.
Life and medical sciences: youth's hurdles recognised
With the rules for the 2008 RAE in place, Anthea Lipsett and Tony Tysome assess the key changes and their impact on departments and individuals
Academics on the research assessment panels for the life and medical sciences have adjusted the initial proposals rules for the RAE to encourage submissions of work from young academics.
The final criteria recognise that junior researchers in universities are likely to submit fewer academic papers than their more experienced colleagues and that they will be of a lower quality.
Sir John Beringer, who chairs RAE Panel D for the biological sciences, pre-clinical and human biological science, agriculture, veterinary and food science, said: "We originally said that new entrants would be expected to have four publications in their subject areas. We have now modified that to say that we recognise that in some cases it might not be necessary.
"In the biological sciences, it is unusual to appoint someone who has not done a couple of postdocs, which would mean they had at least four publications. But there may be rare occasions, such as when someone has been ill or done work of a very high standard, where fewer will be acceptable."
Sir John also warned university departments that the panel would want to see that they have clear plans for staff.
Departments that submit only top-rated staff could be marked down in terms of environment measures, which cover departmental strategies and plans, he said. "That would show that they do not have a continuity plan. We want to see a range of outputs."
It was important for academics on Panel B, for epidemiology and public health, health services research, primary care and other community based clinical subjects, psychiatry, neuroscience and clinical psychology, to clarify the position of trainees who will be expected to submit only two publications.
Clinical lecturers in training grades will not have to submit more than two publications unless they graduate from trainee status before the end of April 2007.
Sir Leszek Borysiewicz, who chaired panel B, said: "Individuals have been given a period of grace because we do not want people to delay in order to fulfil the training criteria for the RAE."
Both Panels D and B recognised there might be rare occasions in their disciplines when a publication's size gave it super-book status. But Sir John said: "We are not encouraging it because we feel it is better to see a wider range of outputs."
The importance of practice-based research, however, has been given full recognition. Sir Leszek said: "We expect most of the submissions we receive will have a potential or real applications in the National Health Service or in industry."