Fears over ‘polishing’ of REF statements spark audit calls

Checking the veracity of narrative-based statements describing universities’ research culture has emerged as a key concern for panellists finalising guidance for the next Research Excellence Framework (REF), Times Higher Education understands.

With the final guidance for REF 2029 set to be published in autumn, panellists and subject experts have been meeting in recent weeks to clarify the rules that will determine how around £2 billion in annual block grant funding will be distributed across the UK sector.

With most guidance on submissions, impact and volume measures already announced, significant attention has focused on the renamed “strategy, people and research environment” (SPRE) section, previously “people, culture and environment” (PCE), of the REF which accounts for 20 per cent of an institution’s overall score.

Alongside discussion of which indicators of research environment excellence should be permitted for inclusion, panellists have also raised the issue of how official documents will be audited and what evidence should be submitted to support excellence claims.

“Approaches to both rigorously evidencing claims and to auditing have been comprehensively discussed and debated by panels and central REF,” one panellist told THE, adding: “We’re absolutely aware of the scrutiny that is warranted.”

That concern relates to the expected narrative-driven nature of the SPRE statements after a PCE pilot found only a handful of research environment metrics on which institutions could be fairly judged without imposing “significant burden” on institutions that, in some cases, would not be able to collect data retrospectively.

“A purely metrics-based exercise would be relatively simple to audit and would offer little opportunity for stretching the truth,” one senior professor at a Russell Group university told THE.

However, narrative statements may tempt universities to embellish their achievements to “present themselves in the best possible light – and, of course, that comes with risks”.

With multiple people working on a submission there is also greater risk that individuals may use artificial intelligence to “help polish the narratives which will lead to indistinguishable submissions that all follow the expected formula”, he added.

“Narrative-heavy work which tends to adhere to a particular writing formula – one where humans would often start by updating and editing a previous submission – is especially likely to be disrupted by large language models,” he continued, stating that the “REF is clearly in this category and it’s the first REF where this has been the case”.

Simon Green, pro vice-chancellor (research and knowledge exchange) at the University of Salford, who was a panel member for the PCE pilot, said he was pleased that panels were discussing auditing because the “tendency to storytell [in narrative statements] is inherent”.

“There is a natural inclination to put the best possible spin on all of your activities but when does that tip over into misrepresentation?” asked Green.

While it was important to establish an audit framework and a mechanism to flag misrepresentations, Green argued it would be “very difficult” to uphold complaints against claims made in narratives.

“Unless the audit finds a very technical breach that has been committed it will be hard to prove [misrepresentation],” he said, adding: “We have to recognise the limits to audit.”

In a statement, a REF spokesman said the “REF team and panels are developing a robust and fair method for assessing and scoring Strategy, People and Research Environment statements to ensure consistency and a level playing field.”

“A robust, proportionate and fully costed audit function is built into the REF 2029 programme,” he added, stating the existing Code of Practice (CoP) guidance had committed to establishing a complaints and investigations process for when individuals believe CoP have not been followed.

“Four complaints were received through the REF 2021 Codes of Practice complaints process, of which two (one partially) were upheld,” he added, noting a report on the process and outcomes, including the upheld complaints, was published.

jack.grove@timeshighereducation.com