TEF 2017: ‘meaningless’ results ‘devoid of credibility’, says v-c

Sir Christopher Snowden, former Universities UK president, attacks ratings in wake of Southampton’s bronze award

June 22, 2017

One of the UK’s most senior vice-chancellors has denounced the teaching excellence framework (TEF) as being “fundamentally flawed” and having “no value or credibility” after his university was awarded bronze.

Sir Christopher Snowden, vice-chancellor of the University of Southampton, made a blistering attack on the TEF as his institution became one of three universities in the Russell Group, alongside the London School of Economics and the University of Liverpool, to be given the lowest rating in the first sector-wide assessment of teaching quality.

Sir Christopher, a former Universities UK president, said the bronze award handed to Southampton – ranked 121st globally in the latest Times Higher Education World University Rankings – was inexplicable given its strong student outcomes and universally positive feedback from TEF assessors.

“There is no logic in our result at all,” Sir Christopher told THE. “How can you have so many positive comments and exceed many of your benchmarks by a colossal margin and still get a bronze?” he said, adding that the university would appeal the result.

Sir Christopher, who led the University of Surrey for 10 years before arriving at Southampton in 2015, said he was particularly mystified by the bronze award as his university had performed strongly on the student satisfaction, graduate earnings and course completion data used to rate institutions.

“I know we have done significantly better [on metric scores] than many gold-rated institutions with a similar student profile – I can show you the evidence,” he said.

That suggested that the TEF standings were heavily influenced by the information contained in a 15-page submission from universities, said Sir Christopher.

“It tells you the way they treated the statements was completely different,” he said. “I know I am not alone in having deep concerns about its subjective assessment, its lack of transparency, and with different benchmarks for each institution removing any sense of equity and equality of assessment.”

The TEF’s assessment of performance against expected benchmark values based on student profiles, rather than absolute outcomes, made it “incredibly easy for some institutions to exceed their benchmarks”, whereas, for Southampton, “the benchmark was so high we could never achieve a positive ‘flag’,” Sir Christopher added.

For instance, Southampton was required to beat its benchmark target of 4.5 per cent for student dropout rates by two percentage points to achieve a positive flag, said Sir Christopher. It did so by just one percentage point.

“It would mean we need a dropout rate of 2.5 per cent – I'm not sure any university in the country could get that,” he said, saying the target was “ludicrous” and that “the benchmarking is fundamentally flawed”.

With universities held to different standards and judged against expected performance, it was analogous to holding a race between “Usain Bolt and a snail”, said Sir Christopher, adding that “you would only need the snail to go a bit faster [than expected] to win as that is how you are judging performance”.

Sir Christopher added that the “concept of gold, silver and bronze was absolutely meaningless” for students seeking educational excellence, while the TEF’s name was misleading as its metrics were not robust measures of teaching quality.

“I think the Competition and Markets Authority may want to have a word with [universities minister] Jo Johnson,” said Sir Christopher. “The TEF has no credibility and I do not even see how league table compilers will use it as it has no value.”

His comments were echoed by other leading UK universities handed a bronze award. Deborah Johnston, pro-director of teaching and learning at Soas, University of London, said the metrics used by the TEF did “not accurately reflect [its] distinctive mission and student body”.

“There is clearly also a London effect with one in three London institutions achieving bronze, compared to just one in eight outside London,” added Professor Johnston, saying metrics “were not benchmarked for London”, where lower retention rates were often due to costs of living.

Julia Black, interim director of the LSE, said its “students’ high attainment and outstanding performance in highly skilled job markets [are] unfortunately…not captured by the TEF metrics”.

And Sir Christopher’s criticisms were echoed by Sorana Vieru, vice-president (higher education) of the National Union of Students, who called the TEF “another meaningless university ranking system…[which] fail[s] to capture anything about teaching quality”.


Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

Reader's comments (8)

Southampton volunteered for this exercise so obviously thought they'd be ok, but now its not gone their way its all a bit like sour grapes. Would he be complaining if the university had got a gold award!? Unlikely
Reading Southampton's submission was interesting, the expectation was for Silver! Talk about not aiming high enough... I wonder just how bad it really is, across the whole UK, most 'lecturers' I have contact with have no 'professional' pedagogical qualification and that perhaps is the root of the problem?
"Southampton volunteered for this exercise so obviously thought they'd be ok" It would be nice if more institutions had boycotted the whole exercise, given concerns raised long before the results, but there are a number of reasons why most did 'volunteer'. Firstly, all participants will be allowed to raise their fees (which haven't increased with inflation since jumping to £9k). Anyone not doing this will therefore be forced to compete with better-funded rivals. Second, it's likely that prospective students, parents, and league table compilers would, in any case, assume the worst of those choosing not to participate.
I wonder whether the exercise would have been quite so meaningless had they been awarded a Gold? It strikes me that if you whinge after the event it seems like sour grapes.
So the 15 page statement for this is meaningless , but getting a professional to write a REF statement because it is an important part is OK...
Leaving aside why the comments might be being made, he does have a point. It's a fatally flawed exercise that does not achieve what it set out to.
Let's be honest, Snowden is just whining because his university did poorly. You can bet your pennies that if he was still at Surrey when the TEF results were released, he would be the first to order the Press team to bombard their social media with their Gold success. And Southampton must be really struggling to beat those TEF benchmarks. After all, the average student comes in with a ridiculously enormous 390 UCAS points - the 36th highest in the country! This only acts to complement its highly diverse social mix of students where 77% of students are from non-working class backgrounds - the 23rd highest in the country. It must be so hard to beat those benchmarks when universities with a higher proportion of middle-class students - Oxbridge, St Andrews, Imperial, Bath - all have more middle class student bodies where the average entrant achieves 480+ UCAS points. Southampton needs to stop making excuses for itself. It's embarrassing, especially when it's coming from a former Universities UK President.
Neiljohn writes ... 'most 'lecturers' I have contact with have no 'professional' pedagogical qualification and that perhaps is the root of the problem?' ... well he must be an HEA supporter to write something as silly as he has. It's the professional pedagogical qualification which is masking poor teaching. After all, if you have one you must be a good teacher!! mustn't you?? Does anybody out there know of an individual that has undertaken a 'professional pedagogical qualification' and NOT been successful?? I know of a significant number of poor teachers (the students are good at letting you know who they are!!) who are festooned with all manner of awards and slaps on the backs. What is laughable about the TEF is the complete absence of viewing individuals ply their trade; just how stupid does it get??