TEF data: so what can we count on?

Scrutiny of datasets informing the teaching excellence framework will ramp up, and rightly so, given the high stakes and risk of misuse

May 26, 2016
Data being analysed under magnifying glass (illustration)
Source: iStock

We live in the age of data. A word that once evoked the deadliest of dead-end jobs (I once spent a month doing data entry in an NHS deanery) now speaks of data scientists who rule the world from trendy cafes, powered by skinny soya lattes.

As the American statistician and management guru W. Edwards Deming put it: “In God we trust, all others bring data”. This is a mantra that’s revolutionising business, dismantling old industries and making billionaires of digital pioneers. It is also, inevitably, finding its way into public policy, and changing the fabric of academia, one of the most data-hungry professions around.

But data must be handled carefully, must be clean and meaningful, and understood and used in the right way – lessons we’re very clear about as the publishers of the Times Higher Education World University Rankings. And when it directly affects the prosperity of individuals or institutions, the integrity of data has to be protected as never before.

This is front of mind in the UK because of the forthcoming teaching excellence framework, which we now know will be tied to university funding (by informing tuition fees) despite intensive lobbying by universities and students to resist such a link.

Last week’s White Paper softened the approach, giving a two year bedding-in period, but we should be in no doubt that more aggressive financial incentives are coming.

One result is that the scrutiny of the data fuelling the TEF will also ramp up, and in our news pages this week we highlight just one of the potential chinks.

Our investigation reveals that the funding council has looked into claims that institutional staff have tried to get undergraduates to inflate their course ratings in the National Student Survey (one of the TEF metrics) on 30 occasions over the past seven years.

This isn’t a huge number in the scheme of things (and only four were serious enough to warrant NSS results being withheld from data sources such as Unistats), but it shows that there’s already pressure to a degree that is warping individuals’ actions.

That pressure will intensify dramatically with the introduction of the TEF.

The reality is that there will always be flaws with datasets of this kind, and the risk of misuse and abuse is equally inevitable.

Most academics will also accept that imperfect data can be useful if handled in the right way (although it has to be said that government has a pretty ropey record in this regard, particularly when it comes to supposedly data-led reform of the NHS).

But while accountability and competition are indeed vital components in universities’ success, there is an underlying worry here: that a toxic culture develops in which everything is measured and the measurement becomes an end in itself.

Deming’s quote about the divine nature of data is counterbalanced by his “seven deadly sins” of management.

Among them are “lack of constancy of purpose”, “evaluation by performance, merit rating, or annual review” and “management by use only of visible figures, with little or no consideration of figures that are unknown or unknowable”.

These are equally important lessons as higher education is re-envisioned.

john.gill@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest