New algorithm hopes to counter potential bias on funding panels

Mathematical model works by trying to remove skewing of results in group funding decisions

February 10, 2017
lawyers and judges arguing
Source: Getty

A new system could be used to rectify the “serious issues” that arise from individuals with varying levels of expertise assessing academic work or grant applications.

The system, developed by mathematicians at the universities of Warwick and Coventry, works by using an algorithm to take into account different scores that each individual on an assessment panel gives a piece of work as well as the competency of each assessor.

It is hoped that the algorithm could be used to make fairer decisions about funding, such as those in the research excellence framework for example, by reducing potential biases.

Many types of decision in academic life are made by groups of people who sit on panels. These decisions could be related to which proposals secure research funding, who is the strongest applicant for an advertised job, or which work gets a four-star rating in the REF, for example.

Often panels dish out the applications for assessment between themselves as there are too many for each person to read them all. So each application may get looked at by only two or three assessors. The scores from each individual assessment are then brought together and averaged so that a joint decision on quality can be made.

But Robert MacKay, professor of mathematics at the University of Warwick, argued that each person will likely assess the same application differently. One person may mark something higher or lower than someone else, and another may be more experienced in the particular field of the application so their opinion should carry more weight. These issues can introduce bias into the averaging process, he told Times Higher Education.

“This [was] a serious issue for example with the [2008] research assessment exercise, and now the REF,” he said. The decisions made during these assessment exercises are important because they dictate, in part, how much funding universities secure.

To get around the potential unreliability of the judgement process, Professor MacKay and colleagues created the algorithm, known as “calibrate with confidence”, that was published in the journal Royal Society Open Science this month. It takes into account the differing standards of each person on an assessment panel and asks panel members to rate the confidence they have in their scoring of each proposal.

As part of their research, the group verified their new method by rerunning a previous assessment of funding proposals. In the original exercise, 44 proposals were assessed with scores ranging from 75 to 87, and 13 proposals secured funding.

Professor MacKay and colleagues found that after using the calibrate with confidence method, only six of the projects that did secure funding would still have been approved, while seven other proposals ranked high enough to secure the money.

The algorithm could act as a “decision support tool” to flag up occasions where potential biases are present, Professor MacKay said.

He added that, so far, the method has not generated any enquiries from the Higher Education Funding Council for England, which administers the REF, but that the mathematics panel of the Engineering and Physical Sciences Research Council had expressed an interest in using it.

holly.else@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Viewed

Most Commented

Lady Margaret Hall, Oxford will host a homeopathy conference next month

Charity says Lady Margaret Hall, Oxford is ‘naive’ to hire out its premises for event

women leapfrog. Vintage

Robert MacIntosh and Kevin O’Gorman offer advice on climbing the career ladder

Woman pulling blind down over an eye
Liz Morrish reflects on why she chose to tackle the failings of the neoliberal academy from the outside
White cliffs of Dover

From Australia to Singapore, David Matthews and John Elmes weigh the pros and cons of likely destinations

Michael Parkin illustration (9 March 2017)

Cramming study into the shortest possible time will impoverish the student experience and drive an even greater wedge between research-enabled permanent staff and the growing underclass of flexible teaching staff, says Tom Cutterham