What should happen to the next research assessment exercise? The Times Higher asks vice-chancellors for their responses
"The research assessment exercise has done the job it was intended for, and the 2008 exercise is an RAE too far. It has already had a damaging influence on universities, including dysfunctional effects on teaching, continuing professional development, internal promotions, staff salaries and inter-institutional transfers. I expect these to continue through to the 2007 deadline with possibly significant 'research star'
transfers taking place cynically and expensively just before the cutoff date. Indeed, there are reports of contracts being signed for appointments beginning in autumn 2007. Abandoning the 2008 RAE now will also ensure we avoid its inevitable iconic status as the last 'real RAE'.
"The recent fears about an alternative system can be overcome by the simple logic that a new judgment system, if credible, must initially give the same results as the old one. We would expect no less of a new assessment system for our students. It is inconceivable that a desired end could be drastic reductions in funding for research in arts and social sciences.
"It is essential that the starting point for a review be clearly defined and take account of the needs of all research areas; fund good research whereverit is carried out; respond to early-career researchers; provide stability yet be responsive to changes on a reasonable timescale; and look primarily at what is proposed in research rather than at history. I would conclude that a simple metrics system is unlikely to be enough and that the dual-support system must be part of the review."
David Melville Kent University
"I have been asking for the 2008 RAE to be scrapped for years. The problem now is that plans are so advanced and, as nobody has tested the robustness of metrics, there is considerable potential for unintended consequences if they alone were used.
"On balance, I would be happy for the sciences and engineering to be entirely metrics based (and recalculated every year). I think for the arts and humanities and for parts (and only parts) of the social sciences some sort of mini RAE would be best. It is important to stress that the overwhelming majority of our QR (quality-related funding - the funding councils' block grants) comes from the science base and therefore calculating that on metrics would save us a huge amount of work."
Anonymous
"We feel that the 2008 RAE should definitely go ahead as planned for five reasons.
"First, many institutions have heavily invested public money in advance of the exercise. It would be a shocking waste were the exercise to be cancelled.
"Second, there is no clear and agreed alternative method of allocating QR.
It would seem sensible to use the 2008 RAE to develop metrics for the areas in which their use is appropriate. But that development will take time; it would be far too late to switch to metrics at this stage.
"Third, metrics simply do not work for the humanities and social sciences, and we need time to develop an alternative method of redistributing the existing pots of QR in these areas.
"Fourth, the RAE serves as an excellent management tool within universities, and it is difficult to see how metrics could replace it, especially in areas where grant getting is not standard practice.
"Fifth, the RAE serves as a measure of quality; the grades are clear statements about research quality, whereas metrics privilege size of activity."
Steve Smith Exeter University
"We are of the firm opinion that a uniform and simultaneous assessment of university research informs the world of the quality and distinctiveness of the UK research base and its institutions. The RAE and the resulting selective funding of the highest quality research has served to promote the international competitiveness of the UK research base. Disruption now, in the absence of a demonstrably better and thought-through alternative, would result in the quality of UK research and institutions not being fully assessed and represented."
Sir Richard Sykes Imperial College London
"I would like the RAE 2008 to be scrapped completely. I've been saying this for some time. I believe the RAE monies should be transferred wholesale to the research councils as additional and full overheads for their grants. I think the Chancellor's metrics method is an unnecessary complication - but he's on the right lines."
Terence Kealey Buckingham University
"We have been systematically planning for RAE 2008 for the past four years to raise the research profile through peer review of a newly founded university that aims to be an institution of quality and distinction.
Moving the goalposts at this stage would be inconsistent and unhelpful to our aspirations."
Mike Saks, pro vice-chancellor, on behalf of David Chiddick Lincoln University
"After 2008, I am not against an approach using metrics so long as it preserves dual funding, is sufficiently robust and recognises that different universities have radically different research portfolios.
"Given the consultation for the next RAE and the momentum already established for 2008, it makes no sense to stall an agreed process without establishing a broad consensus on a replacement. The methodology outlined in (the Chancellor's Next Steps document) is severely flawed and needs lots of additional work to be remotely credible to the academic community."
Paul Wellings Lancaster University
"The costs (both human and financial) of undertaking the RAE outweigh any potential benefits given that the correlation between total research income and Higher Education Funding Council for England funding is so strong.
Since peer review is already built into research funding from competitive sources, there isn't much added value from the additional peer review required for the RAE. If there is a problem for those institutions that focus on arts and humanities, then another means should be found to address their difficulties."
Anonymous
"No intelligent decision can be made on the basis of the information so far available. There are significant financial and reputational implications in any choice, and it isn't sensible to make a simple, swift selection on the basis of back-of-the-envelope calculations.
"The Treasury, the Department for Education and Skills, Hefce and the Office of Science and Technology need first to get their thinking together and then to engage in open and informed discussions with research universities over the objectives and priorities for allocation of research funding for the next ten years."
Anonymous
"If the Eastwood/Wilson committee devises a post-2008 strategy quickly, a very strong reason for still running the 2008 exercise would have to be given. This is, however, a decision that would have to be made in 2006, which is why we are sticking to our presumption (that the 2008 RAE will go ahead).
"We do not share the view of others that the sunk costs of the 2008 RAE represent an investment that cannot be abandoned; we recognise the enormous effort made by academics and the RAE team, but the bulk of the real costs is yet to come. In devising a post-2008 process, the sector should consider that:
* A methodology appropriate for one group of subjects may not be appropriate for another
* A methodology appropriate for evaluating a collection of academics does not have to be appropriate for evaluating an individual
* The use of QR funding calculations to drive allocations in institutions may have to be given up.
"If these propositions are not accepted, then I see little chance of moving away from the RAE process, even post 2008. If these propositions are accepted, then I believe there is a great deal of flexibility available. I would like to see a rolling exercise that runs every year and where any detailed peer review is staggered by unit of assessment (UoA).
"If it is accepted that a rolling exercise can be put in place by all UoA subpanels by 2009, then I would argue that the 2008 RAE would not have to run. The 2001 results could stand with UoA rolling exercises phasing in to replace the 2001 results as they were agreed."
Ian Leslie, pro vice-chancellor, on behalf of Alison Richard Cambridge University
"I would like it to be scrapped altogether. The response that says 'wait until there is something better' is, frankly, bizarre, because we have no idea how the results will be used to gear funding. We are all being asked to shoot at a goal that hasn't yet been located. I agree with Michael Sterling: it's a massive waste of time and resource."
Anonymous
"A results-based system has many attractions, but the devil is in the detail. If the metrics are based on outputs, such as external income as a function of public-sector investment, then there is some logic to moving from a peer-assessment process. But if they are based on inputs of public funds distributed by peer review, such as research council funding, how would such a change be beneficial? I'm open to scrapping the 2008 RAE quickly but only if there is a convincing case to do so."
Michael Brown Liverpool John Moores University
"We think that it is difficult to justify the costs of the 2008 RAE, both centrally and institutionally, given the tiny gain it produces over a metrics-based system. We therefore think that the RAE 2008 should be scrapped for all subjects and replaced by a metrics-based system."
Anonymous
"I was alarmed by the Chancellor's announcement about the RAE as it seemed to be based on not very good evidence about the effects of the exercise and the usual bias in favour of science and engineering.
"The RAE was extensively discussed after the 2001 assessment and Hefce was concerned to ensure that all members of the academic community had an opportunity to comment. It was very clear that, despite a concerted effort to find an alternative, academics see the RAE as the least worst option for the distribution of research funding. In addition, for small institutions such as Northampton that do not have research-intensive missions, the RAE is important for prestige and international quality assurance rather than for any funds."
Ann Tate Northampton University
"At one level, I am indifferent in so far as the RAE, being designed to concentrate resources in a small number of institutions, has become a minor consideration in this university even though active engagement with research remains a strong driver.
"An RAE should be able to spot the nuggets of high quality within disciplines independent of metric and therefore might favour the modest scale or emergent areas more. On balance, I would stick with the RAE one more time, debate the metrics fully and openly (as opposed to the privileged few who seem to have known about this for some time) and then take a balanced view of what to do next, but it would not be a repeat of the RAE as it is."
Anonymous
"This institution has invested a great deal in preparing for the 2008 RAE.
But much of this work has been developmental and will not be wasted if the RAE does not take place. We are strongly of the view that, if the 2008 exercise is to be the last or to have a short shelf life, it will be a wasteful and distracting activity. We would rather put this effort into developing an appropriate successor methodology and advocate an extension of the 2001 RAE distribution model until agreement is reached.
"We obviously want any new methodology to fairly and accurately apply to the arts. A metrics system, which might work for science and engineering, is not acceptable and will severely disadvantage the arts where income figures are relatively small and citations indices do not translate very easily.
"While we support an element of stability and predictability in the funding regime, this should not lead to ossification. It is important to enable the growth and development of research across a broad and diverse community if it is to remain vigorous and competitive."
Michael Bichard University of the Arts London
"It would be strongly counterproductive to abandon the 2008 RAE.
That would create huge destabilisation and waste, bearing in mind the substantial investment of public funds already made by institutions and the extent to which the 2008 RAE is embedded in their current arrangements for managing research.
"There is no consensus on metrics, and it is acknowledged that they do not work for the arts, humanities and social sciences: they constitute a simplistic approach that, if implemented, would kill half of a research base that the Government itself recognises is second only to that of North America."
Andrew Wathey, senior vice-principal, on behalf of Stephen Hill Royal Holloway, London
"We accept that it is likely (and may turn out to be desirable, depending on the system) that the RAE makes greater use of metrics. The difficulties of finding acceptable metrics in relation to third-stream work implies that a proper evaluation will take some time. This cannot be achieved properly by 2007-08.JWe are preparing for the next RAE."
Anonymous
"The current RAE system, despite some weaknesses, has a number of advantages over a metrics-based system. In particular, the grades are an internationally accepted standard of quality, something a metrics-based system cannot provide; and the RAE is a powerful instrument for internal management of research quality and productivity.
"The unintended consequences of a metrics-based system have not yet been explored. For example, a system based on research council grants will inevitably lead to a rapid growth of grant applications to the research councils, shifting the administrative cost of the RAE to them."
Anonymous
"My personal view is that they should do away with the RAE altogether and replace it with a combination of project funding and capacity building. That, of course, sounds idealistic, but the reality is that the current concentration of QR and the concomitant impact on research council funding means that many institutions are denied virtually any capacity funding.
"Given my institution's interest in science and engineering, I am attracted (by the prospect of certain subjects opting out of RAE 2008) because the circumstances for these subjects are different from those of the humanities. It's usually individuals who need support in history and English, whereas science requires a physical infrastructure.
Deian Hopkin, London South Bank University
"The 2008 RAE should not be scrapped. We have invested considerable effort (time and money) in preparing for the exercise.
Although it imposes a considerable burden, we believe that the current 2008 RAE methodology is fair and robust, not least because of a lengthy period of consultation and refinement. We are not aware of any evidence that an alternative form of assessment based on metrics is both fair and applicable across all subject areas. We believe that the 2008 RAE offers a unique opportunity to trial any new (metrics-based) system.
"We do not believe certain subjects should be able to opt out of the 2008 RAE because:
* We fail to see how the 2008 RAE and the new metrics-based systems can exist side by side
* We do not accept that a metrics-based assessment can be applied blindly to any subject area without a great deal of preparatory work to establishJthat the quality of such a method of assessment matches that of the 2008 RAE.
"We see no way of achieving this without including a large element of tried-and-tested peer review. Any new system that has the effect of concentrating resources in a few large institutions in which science, technology and medical research predominate is simply unacceptable."
James Stirling, pro vice-chancellor, on behalf of Kenneth Calman Durham University
"All the 2008 RAE panels should be given greater freedom and encouragement to use metrics, which could inform the three elements of assessment to the degree they judge appropriate, but they should not move away from the principles that outputs, environment and esteem are being assessed.
"Hefce should ask a group of modellers and statisticians to look at running a totally metrics-based assessment alongside the RAE to try to mimic the outcomes. This might pave the way for a metrics-based future QR distribution on possibly an annually updated basis."
Anonymous
‘I cannot believe it is being suggested at this stage that the RAE should be abandoned. Changing the rules of the game now is incomprehensible’
Graham Upton, Oxford Brookes
‘There’ll be no forward looking or integrity in these replies. They will be largely motivated by how much loot you think you’re going to get from each system’
Peter Knight, UCE
‘The effect would be to sabotage high-quality research in smaller universities and undermine funding of the arts and social sciences’
Chris Snowden, Surrey
‘RAE 2008 should give us time to work out the correct form of metrics. If that could be done for sciences, engineering and medicine, I would be happy to change sooner for these disciplines’
Michael Arthur, Leeds