OfS pushes in-person exams to ‘protect sector’s reputation’

Universities are being urged to consider returning to in-person and closed-book exams by the English regulator as part of its new guidance on how to maintain “valid and reliable” assessment in the age of artificial intelligence.

The Office for Students (OfS) has for the first time detailed institutional approaches to AI that might prompt regulatory intervention, promising to “hold institutions to account when they fall short”.

The updated advice responds to the findings of a four-month study by the OfS and Advance HE that finds AI use to be widespread among students and university employees.

The Making AI Work for Students report describes a need for greater “literacy” about how to use technology effectively, ethically and in compliance with institutional standards and expectations.

It also identified an “appetite for greater clarity about how OfS regulation applies in the context of AI”, the regulator said, prompting the agency to outline examples of approaches that are likely to comply with its quality conditions.

Under condition B4, which mandates the use of effective assessment methods, higher education providers are already expected to prepare “secure” processes for testing students’ learning.

The OfS said this means running “secure” assessments or those “which are designed to prevent the use of AI”, suggesting the use of “in-person examinations, presentations, vivas (oral examinations), performance-based assessments, and assessments that require students to demonstrate their knowledge and skills through practical or applied tasks”.

While stopping short of mandating AI-free assessments, the regulator believes these will often work best. Where AI use is permitted in exams this should be set out clearly “through realistic, meaningful and transparent academic regulations and accessible policy and guidance”.

Josh Fleming, director of strategy and delivery at the OfS, called AI a “generational risk to the system as it stands, threatening to undermine students’ learning and institutions’ ability to assess them”.

“We need to see far greater efforts to tackle these risks, including far greater use of AI-free assessment,” he added.

The report was based on conversations with some 300 students and staff members based at more than 120 institutions.

Crucially, it finds that AI poses “significant risks” to assessment’s validity, threatening the reputation of the higher education sector.

“A significant proportion of assessments used in the sector remain in unsupervised, non-controlled formats where AI can be accessed by students completing them,” the report finds.

“Student research participants frequently told us that students they knew had completed assessments using AI, sometimes in just a few minutes, and received a similar grade to those who had completed the assessment themselves.”

Some of the students the regulator spoke to “said explicitly they believe the assessments run by their institution are not fit for purpose or do not accurately measure students’ knowledge and skills”.

Pointing to its new quality framework, announced last month, the OfS said universities failing to comply with its expectations on AI will face sanctions. Concerns identified from university submissions, provided as part of the process, or “other regulatory intelligence” could lead to the “provider receiving a lower award and could be used to inform a decision over whether to conduct a further investigation of the quality of provision at the provider”, the regulator said. 

The research also suggests that staff and students might not see eye to eye when it comes to AI’s use in marking and feedback.

According to the findings, students in the UK are “often critical” of the technology’s use in marking, with some describing feedback from “expert teaching staff” as key to higher education’s “value proposition”.

But teaching staff told researchers that AI-generated feedback could be a “beneficial starting point” that could help to free them to spend more time actually interacting with their students.

Speaking in Parliament the day before the report’s release, Simon Banks, a vice-dean for education and professor of natural sciences at King’s College London, told politicians that universities needed to be able to certify their students “with confidence” and to “be able to stand behind that certification”.

He said there was a “growing, wide acceptance” of the fact that some assessment methods are “seriously threatened” by generative AI’s capabilities.

“We are going to have to bring a lot more assessment directly back in [to institutions],” Banks said.

“And we need to do that in a way that doesn’t lose the benefits that were genuine benefits of those remote assessments.”

Banks said “there will absolutely be a place for examinations”, oral exams and a “suite” of types of assessment – but highlighted the need to reflect different educational needs in the process.

“That’s got to be done in a way that is designing in the inclusivity from the start,” he said. “You can’t retrofit that afterwards.”

georgia.luckhurst@timeshighereducation.com