AI marking trial ‘not looking to replace humans’

A UK trial of artificial intelligence marking tools is not looking to replace academics, organisers have insisted, despite staff concerns.

Jisc, a non-profit organisation that supports universities with their digital infrastructure, is midway through a year-long pilot that aims to assess how new software could be utilised by universities.

Since last autumn, 10 universities have been trialling the Graide platform, which creates AI feedback and marks based on assessment rubrics and example essays, before it is confirmed by academics and returned to students.

A further five institutions are also trialling TeacherMatic, which helps academics create resources, including learning plans and coursework tasks.

Participating institutions have to inform their students that they are part of the trial and that their assessments are being marked by AI, while some universities have opted for an “active consent” model where students can opt out, and others are using the tech to mark historical student assessments, explained Tom Moule, senior AI specialist at Jisc.

Academics must “always be in the loop” and “these tools shouldn’t be used as a final marker – humans should always be reviewing the feedback”, he said.

Moule said the trial is especially “interested in understanding both the efficacy of AI in supporting marking and feedback, particularly its role in reducing workload”, and insisted it is not about replacing staff. 

“One of the big promises of AI across all sectors and parts of society is its impact on productivity and supporting people to do more work,” said Moule.

“We know that in the context of tertiary education, one of the real pain points around workload is the assessment process. We also know that one of the pieces of feedback that is consistently given by students is around their perception of the shortcomings around marking and feedback – whether that’s around consistency, the timeliness, the volume, or the amount of feedback.”

Staff have long had concerns about the creeping presence of AI in university processes.

One academic who raised concerns about the trial with Times Higher Education said that the software could have “significant implications for staffing”. They also said the trial could result in “the devaluation of marking as a core element of our work”, having wider ramifications for academics.

“Marking and assessment is integral to the learning experience, and taking time over it is part of our obligation to students,” they said.

Moule said Jisc is “very aware” that of the sensitivities surrounding the trial and that this is a “high stakes” area. But he insisted that the results have so far shown the importance of educators in the marking process. 

He said that student feedback collected to date has shown that they value feedback from academics, and that AI can supplement, rather than replace, this.

“I think what we’re finding is that the more people are learning about the attitude surrounding the pilot and also the tools, they are seeing that this question of whether they will be used to just cut humans out the loop and facilitate a displacement of staff is not the case,” Moule said.

One of the greatest challenges of introducing AI into the assessment process, he said, is the tension between trying to save time and “ensuring that humans are very much firmly in the loop”, noting that “the more oversight you put in, the more time you eat into those time savings”. 

Therefore, Moule believes that AI playing a role in summative assessment is “a long way off”, and that it is better suited to providing more frequent feedback on formative assessments.

AI tools could be used to give feedback on a first draft of a student’s essay, or even a dissertation. It could also be used to write up detailed feedback for students based on “rough notes” written by academics.

“Without resolving that tension, that is going to make it very difficult for this to be commonplace in summative assessments and the high-stakes assessments that contribute significantly to the qualifications that are conferred on students,” he said.

But overall, despite staff reservations, Moule believes that the trial shows the significance of the “very important relationship between student and teachers”.

“We’re seeing that the more we can show people the tools and talk about the implications and the insights that we’ve developed from the stakeholders perspective on what students want…[the more] we see there’s a real role for educators there.”

juliette.rowsell@timeshighereducation.com