Reputation risks of lotteries for research grants inhibit funders

Benefits of innovative grant selection method must be balanced with risks to funders’ reputations, explain research leaders

December 16, 2021
Woman drawing a lottery with cathedral in the background, research grants
Source: Shutterstock

Awarding research grant funding using lotteries will lead to only “negligible” savings but could expose funders to considerable reputational risks, funding chiefs who have pioneered such systems have admitted.

With several major European research funders expected to announce experiments with grant lotteries next year, there is growing interest in how partial randomisation of grant selection – in which funding applications that pass a certain quality threshold are chosen at random – will work.

One potential benefit of such modified lotteries, which have been trialled in New ZealandGermany, Austria and Switzerland in recent years, is that they may remove the need for lengthy discussions on the merits of shortlisted entries, resulting in efficiency gains.

But a study by the Research on Research Institute, which interviewed leaders from six funders who had piloted grant selection lotteries, found that “there was little certainty about its money saving properties and participants who had implemented the method felt that the amount of time saved was negligible”.

“It doesn’t save a lot of time,” explained one interviewee who said that it “might shorten panel meetings, because it relieves the reviewers [of the need] to assess very marginal or non-existing differences among top applications”, but this was a “tiny advantage”.

One research leader claimed that setting up randomisation was more expensive because peer reviewers took longer to identify projects to place into the lottery rather than just selecting the very best applications. Reviewers also had the time-consuming task of identifying projects that were “so bad that it would actually damage the reputation of [our organisation] if they would be drawn by lot”, he added.

That reputational risk of funding a terrible project through the novel lottery system was one of the biggest concerns for funders who feared that the chance nature of selection would expose them to allegations of incompetence, the study continues.

One research leader explained his fear that negative newspaper headlines could lead to a perception that “our organisation isn’t capable of doing what it is supposed to do”.

“We don’t want the newspapers to say the organisation is now gambling with the research funding,” said another interviewee about this “nightmare” scenario.

James Wilsdon, professor of research policy at the University of Sheffield, who carried out the study with Helen Buckley Woods, a research associate at Sheffield, said funders worried that lotteries implicitly acknowledged the “huge amount of randomness” that the selection of grants via traditional means often involved.

“The fear is lotteries are ceding the ground of the expert – from which the authority of funders to award money comes – and admitting there is a huge amount of randomness in who gets money,” said Professor Wilsdon, adding that the introduction of lotteries required an “epistemic humility about claims to excellence within the system” that some funders found difficult.

For most funders, the capacity of lotteries to increase fairness in decision-making by removing the possibility for bias, which may also improve the diversity of researchers chosen, was a major positive; while others felt that riskier projects with a potentially higher pay-off were also more likely to be funded.

“I’m convinced it increases fairness,” explained one research leader. “It accommodates the random or the chance element which we know exists in these panel processes.”


Print headline: Chance brings ‘negligible’ benefits

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

Reader's comments (3)

Lotteries are not the answer to current difficulties in the system. There are quite enough random elements in life already so another one should not be added. The use of more anonymised applications and 1-page or 2-page initial application stages would be much better. However, a real difference would be made if there was more attention paid to value for money. Those of us who are cheap to run lose out to the big hitters in terms of grant income because the system only seems to value money brought in rather than what happens to it.
I like the phrase "Those that pass a certain quality threshold". So rather than replacing any element of the cumbersome element of apply, apply, apply, this rationing by lottery will just add to the hoops that applicants must jump through. If we keep 'rationing by application' at ;east we maintain some quality control'. If we move to 'rationing by lottery' at least we free up some time spent on those applications. This system seems to retain the worst of both methods.
Many funding decisions are seemingly random anyway, with very highly rated proposals unfunded. At least this system would be honest rather than pretending that one can accurately compare the quality of proposals in different fields.