Survey flaws

April 13, 2017

While readers should be encouraged by some of the changes made to the Times Higher Education Student Experience Survey (23 March), it remains deeply flawed. It is honest that you admit (even if only in the small print) that a difference in score of about 3.7 would be needed to claim that two institutions’ scores are statistically significantly different (so a university ranked 61st might not have a significantly different score from one ranked 22nd or one ranked 94th). However, the survey suffers from many of the problems of all such surveys: arbitrary weightings (which don’t in fact match the influence of the item on the final score), arbitrary choice of questions (why ask about sports facilities but not art, music or theatre facilities?) and so on.

The decision to report subsets of responses as different tables is also to be applauded, but there is little evidence that the factors chosen make statistical or theoretical sense. There is evidence that some questions are institution-level questions (such as “good community atmosphere”) while others are at the department level (such as “well-structured courses”), but not that there is any structure beyond these factors.

The key problem, however, is that there is a strong correlation between the response rate and the rating (even after accounting for institution size): that is, institutions that have proportionately more students participating in the survey get higher ratings. Until there is a clear explanation of why this should be the case and why there is a curiously high number of institutions with exactly 100 respondents, we cannot make sense of the results nor trust the rankings.

Adrian Simpson
Principal, Josephine Butler College
Durham University

Send to

Letters should be sent to:
Letters for publication in Times Higher Education should arrive by 9am Monday.
View terms and conditions.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Featured Jobs

Technical Officer (Paramedic)

Staffordshire University

Professor in Marketing

Henley Business School

Lecturer or Senior Lecturer in Social Work

University Of The West Of Scotland

Research Service Manager

London School Of Hygiene & Tropical Medicine (lshtm)
See all jobs

Most Commented

Daniel Mitchell illustration (29 June 2017)

Academics who think they can do the work of professional staff better than professional staff themselves are not showing the kind of respect they expect from others

James Fryer illustration (27 July 2017)

It is not Luddism to be cautious about destroying an academic publishing industry that has served us well, says Marilyn Deegan

As the pay of BBC on-air talent is revealed, one academic comes clean about his salary

Senior academics at Teesside University put at risk of redundancy as summer break gets under way

Capsized woman and boat

Early career academics can be left to sink or swim when navigating the choppy waters of learning scholarly writing. Helen Sword says a more formal, communal approach can help everyone, especially women