Survey flaws

April 13, 2017

While readers should be encouraged by some of the changes made to the Times Higher Education Student Experience Survey (23 March), it remains deeply flawed. It is honest that you admit (even if only in the small print) that a difference in score of about 3.7 would be needed to claim that two institutions’ scores are statistically significantly different (so a university ranked 61st might not have a significantly different score from one ranked 22nd or one ranked 94th). However, the survey suffers from many of the problems of all such surveys: arbitrary weightings (which don’t in fact match the influence of the item on the final score), arbitrary choice of questions (why ask about sports facilities but not art, music or theatre facilities?) and so on.

The decision to report subsets of responses as different tables is also to be applauded, but there is little evidence that the factors chosen make statistical or theoretical sense. There is evidence that some questions are institution-level questions (such as “good community atmosphere”) while others are at the department level (such as “well-structured courses”), but not that there is any structure beyond these factors.

The key problem, however, is that there is a strong correlation between the response rate and the rating (even after accounting for institution size): that is, institutions that have proportionately more students participating in the survey get higher ratings. Until there is a clear explanation of why this should be the case and why there is a curiously high number of institutions with exactly 100 respondents, we cannot make sense of the results nor trust the rankings.

Adrian Simpson
Principal, Josephine Butler College
Durham University

Send to

Letters should be sent to:
Letters for publication in Times Higher Education should arrive by 9am Monday.
View terms and conditions.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Most Commented

Felipe Fernández-Armesto takes issue with a claim that the EU has been playing the sovereignty card in Brexit negotiations

Female professor

New data show proportion of professors who are women has declined at some institutions

John McEnroe arguing with umpire. Tennis

Robert MacIntosh and Kevin O’Gorman explain how to negotiate your annual performance and development review

Man throwing axes

UCU attacks plans to cut 171 posts, but university denies Brexit 'the reason'

opinion illustration

Eliminating cheating services, even if it were possible, would do nothing to address students’ and universities’ lack of interest in learning, says Stuart Macdonald