Measures to prevent gaming of Australia’s research assessment exercise have been welcomed, but questions have been raised about why it will take years to introduce them.
The Australian Research Council has promised to implement all recommendations from the review of Excellence for Research in Australia (ERA), including proposals to stop universities “manipulating” the system. Under one change, the “census date” approach – where a publication can be evaluated so long as its author worked for the university on a specified date – will be replaced by a “byline” requirement, where submitted papers must carry the university’s name.
Nicholas Fisk, deputy vice-chancellor (research) at UNSW Sydney, said that this would discourage universities from hiring “big-hitters” just before the census date. He said that ERA had encouraged a culture of “buying in kings and moving on your pawns”.
“Someone arrives at five to midnight, and you get six years of publications from them. There’s nothing wrong with buying in quality or moving some people on. But those decisions should be taken strategically. They shouldn’t be driven by ERA.”
Professor Fisk said that jettisoning the census date and other proposed efficiencies would ease the “huge burden” of ERA-related data collection that cost his institution upwards of A$2 million (£1.1 million) per round. Proposals to automate data collection would also discourage universities from selectively categorising their publications to “optimise” their ratings.
Allocating a publication correctly might not affect how a field rated, but allocating it elsewhere could tip a closely related field past the benchmark for above or well above world standard.
The switch away from census dates would improve the assessment’s rigour while saving universities time and money, Professor Fisk said. But the review proposed making the change after the next ERA round in 2023. “The big question is, why not now?” he asked.
The review recommends a delayed approach to avoid “unintended consequences” and to think through “complexities”, such as how to treat creative works and other “non-traditional research outputs” that do not carry bylines.
But the report urges quick implementation of a new rule allowing ERA evaluators to overlook “miscoded” papers in universities’ submissions. And the ARC says that it will immediately work on arrangements to publish the “metadata” for research submissions.
This would allay a “longstanding concern” that ERA does not disclose how much research universities submit. “Two institutions can receive an identical ERA rating [although] volume of activity may vary markedly,” the report notes.
The Group of Eight universities highlighted this shortcoming in a submission to the review. “It is not just excellence that should be rewarded but excellence at scale,” it pointed out. “Scale…demonstrates consistent capability.”
Chief executive Vicki Thomson endorsed the new measures. “It is critical that ERA [shows] both where Australia has the highest level of research excellence and the volume of that excellence,” she said.
Angel Calderon, principal planning and research adviser at RMIT University, said that metadata would improve transparency. But gaming would “still happen”, because universities could pick and choose what they submitted.
Universities might not forward research deemed uncompetitive or commercial-in-confidence, for example. “If we want to truly measure the impact universities have, all outputs need to be considered,” he said.
Professor Fisk criticised the review’s recommendation to conduct assessments every three years, although many submissions had favoured a five-year cycle. “The UK does it every seven years, and you don’t hear screams from Britain to do it more frequently,” he said.
