Survey flaws

April 13, 2017

While readers should be encouraged by some of the changes made to the Times Higher Education Student Experience Survey (23 March), it remains deeply flawed. It is honest that you admit (even if only in the small print) that a difference in score of about 3.7 would be needed to claim that two institutions’ scores are statistically significantly different (so a university ranked 61st might not have a significantly different score from one ranked 22nd or one ranked 94th). However, the survey suffers from many of the problems of all such surveys: arbitrary weightings (which don’t in fact match the influence of the item on the final score), arbitrary choice of questions (why ask about sports facilities but not art, music or theatre facilities?) and so on.

The decision to report subsets of responses as different tables is also to be applauded, but there is little evidence that the factors chosen make statistical or theoretical sense. There is evidence that some questions are institution-level questions (such as “good community atmosphere”) while others are at the department level (such as “well-structured courses”), but not that there is any structure beyond these factors.

The key problem, however, is that there is a strong correlation between the response rate and the rating (even after accounting for institution size): that is, institutions that have proportionately more students participating in the survey get higher ratings. Until there is a clear explanation of why this should be the case and why there is a curiously high number of institutions with exactly 100 respondents, we cannot make sense of the results nor trust the rankings.

Adrian Simpson
Principal, Josephine Butler College
Durham University


Send to

Letters should be sent to: THE.Letters@tesglobal.com
Letters for publication in Times Higher Education should arrive by 9am Monday.
View terms and conditions.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

men in office with feet on desk. Vintage

Three-quarters of respondents are dissatisfied with the people running their institutions

students use laptops

Researchers say students who use computers score half a grade lower than those who write notes

Canal houses, Amsterdam, Netherlands

All three of England’s for-profit universities owned in Netherlands

Mitch Blunt illustration (23 March 2017)

Without more conservative perspectives in the academy, lawmakers will increasingly ignore and potentially defund social science, says Musa al-Gharbi

A face made of numbers looks over a university campus

From personalising tuition to performance management, the use of data is increasingly driving how institutions operate