Poor students’ ‘cognitive gain’ may play role in TEF

Debate may also consider efficacy of running new framework in tandem with REF

July 9, 2015
Two bulls with locked horns
Source: Getty
Number crunch: the TEF may measure ‘journey travelled’ and whether disadvantaged students get jobs and see real improvement

Metrics including “standardised tests” for students as well as measures of the number of disadvantaged students admitted to universities and their “journey travelled” are under consideration for the teaching excellence framework.

Other key issues to be determined in the months of debate ahead on the planned TEF for higher education institutions in England include whether the framework will be a continuous exercise or take place at a point in time that could even be alongside the research excellence framework.

Jo Johnson, the universities and science minister, announced his plan to introduce a TEF, which would include “outcome-focused” metrics, to drive up university teaching standards in a speech last week. It is thought that a timetable for introduction of the TEF has been discussed by government but is yet to be settled.

The government is expected to set up a series of advisory groups to feed into the autumn Green Paper on the TEF announced by Mr Johnson. The Green Paper is likely to set out principles rather than an exact framework, and would be followed by a formal consultation.

The framework itself would then be developed and piloted – but sources expect pilots to happen “sooner rather than later”.

The question of what metrics will go into the TEF remains to be decided, but the government has some clear priorities already.

David Willetts, the former universities and science minister, regarded by some as the source of the idea for a TEF, said that his personal view was that “quite an eclectic mix” of metrics would be needed.

One key area, likely to prove controversial for universities, will be how the TEF assesses the “learning gain” made by their students during their courses.

The Higher Education Funding Council for England has commissioned a number of pilots of projects to measure “learning gain”, the results of which will be considered in planning for the TEF.

Mr Willetts referred to the Collegiate Learning Assessment used by more than 200 higher education institutions in the US.

He said that this involved “fairly basic cognitive tests” administered to students “on a survey basis in their first term and their final term”.

He added: “I don’t think it should be beyond the wit of man to get some kind of measure of cognitive gain like that [in England].”

Hefce states on its website that the learning gain pilots will look at a range of possible methods including “standardised tests”, which would measure “the acquisition of certain skills” and could be “administered to students either as part of their formative or summative assessment for their degree or as an additional exercise alongside the course”. The tests could be “discipline-specific, or focus on generic skills”.

Other methods to assess learning gain to be examined in the pilots include “measuring the progress in students’ achievement by comparing the difference between grades at two points in time”. This could, Hefce states, include “comparing the difference between actual grades at two points in time, using a standardised measure of these grades (such as the grade point average)”.

Mr Johnson called for universities to move to a GPA system in his speech.

The minister also identified four aims for the TEF, including “to recognise those institutions that do the most to welcome students from a range of backgrounds and support their retention and progression to further study or a graduate job”.

There are suggestions that the government may consider including in the TEF a metric on the proportion of disadvantaged students admitted by each institution.

Mr Willetts said: “What we are really after is quite simply a measure of journey travelled. If a university is taking students predominantly from disadvantaged backgrounds with lower A levels and is then getting them into graduate jobs and giving them real cognitive gain, that achievement is not caught by current league tables, which are entirely focused on research and prior attainment.

“What I’m sure Jo is after, and what I believe in, is some measures that capture universities as doing well at that challenge of really accelerating people’s intellectual development.”

Hefce is already piloting new questions on “student engagement” for the National Student Survey, also seen as likely to play a part in the TEF.


By the numbers: metrics-based approach to TEF must be carefully considered

The architects of the teaching excellence framework should be wary of uncritically adopting a metrics-driven approach, according to the chair of a major review of the use of quantitative indicators in research.

The independent review of the role of metrics in research assessment and management, commissioned last year by David Willetts when he was minister for universities and science, concludes that although metrics have a place, they need to be used “responsibly” (see this week's feature).

In relation to the research excellence framework, the review endorses the use of metrics as background information for assessors but does not recommend any significantly greater role for them in the next exercise, likely to occur in 2020.

The review’s chair, James Wilsdon, professor of science and democracy at the University of Sussex, said that he had been struck by the fact that in the week before the review reported, the current universities and science minister, Jo Johnson, had set out plans for a TEF based on “a clear set of outcome-focused criteria and metrics…underpinned by an external assessment process undertaken by an independent quality body”.

Professor Wilsdon said that many teaching metrics were “more well established and…less controversial in terms of their inherent construction” than those used in research. But he still worried that the TEF was “being presented unproblematically as if it should include metrics” and called for the debates about the principles underpinning the TEF and the REF to be “joined up”.

He said that the “main warning” of his review was that “every indicator brings with it inbuilt assumptions, biases and politics” that will “push the sector in one way or another” because of universities’ “strategic responses” to them. Hence, although metrics “aren’t inherently a bad idea, we do need to think through carefully which ones are being selected, how they are being weighted and the additional pressures, burdens and positive incentives they will create”.

Professor Wilsdon was also troubled by Mr Johnson’s remarks, in his speech at Universities UK on 1 July, that he had “no intention of replicating the individual and institutional burdens of the REF” and that the TEF would be “proportionate and light touch, not big, bossy and bureaucratic”.

“That is not exactly a glowing endorsement of the REF,” Professor Wilsdon said. But, in his view, the results of the REF largely commanded confidence and were vital to convincing the government that quality-related funding was being well spent.

This was particularly crucial because it was not a “given” that the dual support system for research would survive the next spending review – particularly in light of the Nurse Review’s focus on the research councils.

Paul Jump

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Reader's comments (2)

Ok then, when's this TEF coming in? The Budget says it will be in effect in 2017/18. That's ages away, surely. Well no. If the fees are to be varied by universities who have 'passed' the TEF, then its outcomes are going to have to be known pretty early in 2016/17. Remember the requirements of the CMA - how could we have a situation with students applying unable to tell what the fee would be? Data crunching takes a while - even in a big data world. You'd have to have the data in from 2015/16. That's next year. Whatever data they think they're going to use, it's going to have to already be in existence in 2015/16. Take a look at the length of time it takes to shift HESA data collection - that would take at least 2 years to get in place. So anything in the TEF has got to use data that we are already planning to collect in 2015/16. Even David Willett's cheerful testing notion - even if he really thinks we can assess the 'gain' in cognitive skills over a degree course, that data won't be available for 2017/18 - If you tested new students next year they'd still be on courses in 2017/18. (US comparisions are unhelpful - SATS & GRE lay a foundation for a more apptitude based testing regime because of the continued approach to general education in UG degrees). BIS must have a really cunning plan - they've let the Treasury use the TEF in the budget - so I really look forward to the Green paper that explains it...
"So anything in the TEF has got to use data that we are already planning to collect in 2015/16." My prediction is that one of the key 'output measures' will be based on the data on the income of graduates who studied under the variable (aka £9000) fees, whose salaries will be known in April 2016 when they start repaying the student loans. For example, this could be the percentage of graduates who earn in excess of the £21,000 threshold for commencing repaying the student loan. This has the advantage that the RAB charge for each university (and each programme within each university) can be calculated in advance.