NSS manipulation claims ‘raise questions’ about data reliability

Allegations of ‘inappropriate influence’ emerge on survey that will be key TEF metric

五月 26, 2016
Construction crew dismantle a building, Shanghai, China
Source: Getty
Behind the facade: there are worries that ‘yea-saying’ by students in the NSS is helping to give institutions flattering scores

Allegations that universities have attempted to inappropriately influence responses to the National Student Survey have been revealed – including staff filling in responses – raising questions about the exercise’s planned role in the teaching excellence framework.

The Higher Education Funding Council for England, which is responsible for coordinating the questionnaire across the UK, has investigated claims that institutional staff have tried to get undergraduates to inflate their course ratings on 30 occasions over the past seven years.

The data, released to Times Higher Education under a Freedom of Information request, were described by researchers as likely being a snapshot of a wider problem, the result of intense pressure on staff to help institutions maintain their position in league tables that draw on the NSS.

In most cases, the alleged interference focused on inappropriate influence via email, briefing or other communication: for example, reminding students of the impact that the NSS has on employers’ perceptions of institutions and, therefore, on their job prospects as graduates. But in six instances, staff were suspected of being directly involved in completing NSS responses.

Of the 27 investigations that have concluded, 17 resulted in institutions being asked to take specific actions, and there was insufficient evidence in six others. In four cases, a panel convened by Hefce judged that, on the balance of probabilities, a course’s NSS data were unreliable, and the results were suppressed on sources such as the Unistats website.

There is also evidence that allegations of inappropriate influence are being made more frequently, with all but 12 of the complaints having been lodged since 2014.

Camille Kandiko Howson, senior lecturer in higher education at King’s College London, said that this was concerning.

“I still think that the sector can have confidence in the data, but I think there is a worrying trend,” she said. “If, as time goes on, people become more aware and start to report more activities, it may be that this is a more extensive practice than we have been made aware of previously.”

Concerns for ‘integrity’ of data

Details of the allegations emerged in the wake of the government’s decision to confirm plans to use responses to NSS questions as a key metric in the TEF, meaning that students’ responses could ultimately influence the tuition fees that an institution is allowed to charge.

The allegations of inappropriate influence can be seen alongside concern about the growing phenomenon of “yea-saying” in the NSS, whereby respondents give the same – usually positive – response to every question. In the 2014 survey, 6.1 per cent of those completing the NSS online did this, up from 1 per cent in 2005, and it has been suggested that “overzealous promotion” by universities concerned about their public profile may be to blame.

Dr Kandiko Howson also highlighted the recent vote by the National Union of Students to either sabotage or boycott the survey, in protest against plans for the TEF, and noted that this action, combined with the other issues, “does raise questions about the overall integrity of the data”.

Questions have also been raised about Hefce’s investigative capabilities, with the funding council confirming that its usual response to allegations is to ask institutions in question to conduct an investigation themselves, and to order an independent probe only if it is not satisfied with the provider’s approach.

Dr Kandiko Howson said that the introduction of the fee link via the TEF would likely “require more substantial action" to be taken, "given the importance of the outcomes moving forward”.

A Hefce spokesman argued that the number of cases where data had been suppressed was “very small” compared with the hundreds of thousands of students and hundreds of institutions that take part in the NSS every year, and added that the funding council took its role in protecting the robustness of the NSS “seriously”, issuing guidance to institutions and regularly checking data for evidence of inappropriate influence.

“Students and the wider public can be reassured that we take clear measures to protect the robustness of the NSS,” the spokesman said.

“Given that Hefce receives just a handful of allegations each year and these are related to specific courses, rather than institution-wide, there is no evidence to suggest either that there is a wide-scale problem with inappropriate influence or that Hefce’s process of investigating allegations is not effective or fit for purpose.”


Releasing names ‘not in public interest’

The names of universities accused of exerting inappropriate influence in the National Student Survey have been kept secret.

Times Higher Education argued that it was in the public interest for the identities of the institutions involved – or at least those where the allegations were serious enough for results to be suppressed – to be released.

But Madeleine Atkins, chief executive of the Higher Education Funding Council for England, ruled that the public interest lay with allowing allegations to be investigated and responded to “without being inhibited by the public gaze”.

In a letter to THE, Professor Atkins says that Hefce relies on institutions to investigate allegations of inappropriate influence internally and to report the outcome to the funding council, and that publicity could hamper this process.

She says that all the cases have been related to a specific course, rather than a whole institution, and that as such releasing the names would have “a disproportionate effect” on a university’s reputation.

THE appealed against the ruling, highlighting the increasing significance of NSS results in the reputation and recruitment efforts of institutions, and also their forthcoming role in the teaching excellence framework.

THE also argued that increased openness might encourage more students to report concerns that they have about completion of the NSS.

However, the appeal was rejected. A Hefce spokesman said that the decision was taken “to protect the student interest and the robustness of the NSS”.

“When allegations of inappropriate influence are raised, institutions are requested to undertake an internal investigation and report their findings, which in turn are considered by a Hefce panel,” the spokesman said. “Consequently, the process relies on institutions to be frank and candid in their response so that appropriate action can be taken.

“Hefce’s ability to respond to allegations would be hampered if institutions were named because they would be less forthcoming in the information provided to the panel.”


Print headline: Can NSS data be trusted amid claims of cheating?



  • 注册是免费的,而且十分便捷
  • 注册成功后,您每月可免费阅读3篇文章
  • 订阅我们的邮件
Please 登录 or 注册 to read this article.

Reader's comments (2)

Schools and colleges have been judged on this criteria for years and everyone has paid attention and matters have improved. There is simply bad teaching in universities, I have found that myself and have reported the misuse of statistics. In my case only a small percentage of postgraduates completed the survey it was well below 10% nevertheless everywhere on campus the statement was that 71% of students were satisfied with the teaching. This is misleading advertising by the university. In my case the School of Law failed to follow university guidelines and failed to follow the rules in the School of Law's own handbook. This is a School of Law for heavens sake. It is right to point out the appalling cavalier approach to teaching and learning by certain staff. Fee payers deserve better.
I am reminded of the FACT that Kingston University's Psychology Department was caught red-handed pressuring students to falsify their responses to the NSS. A contemporaneous set of recordings was made by a number of students, who ultimately passed the evidence to me, whereupon I released it publicly on my website and via Wikileaks. The Department was removed from the set of results for that year by HEFCE following a Public Interest Disclosure and HEFCE investigation. I subsequently learned that the VC at the time was infuriated at the then Dean of the Faculty where Psychology resides for, in effect, having gotten caught in the act of promoting such a practice to staff, and this Dean was forced to take early "retirement." Whereas the VC publicly and disingenuously denied that this was a widespread policy within the Faculty or University. https://wikileaks.org/wiki/Kingston_University_National_Student_Survey_fraud_recording The bottom line is that UK universities and HEFCE have appeared to have learned little by way of preventing fraud in the implementation of the NSS, thereby calling into question the entire system of ranking into league tables. It seems that universities are ranked in this respect based, in part, on how well they get away with fraudulently influencing the results, rather than based on any genuine assessment of quality or student satisfaction.