New way to measure graduate success ‘still just a snapshot’

The Office for Students’ (OfS) proposed changes to how it measures graduate employment might succeed in convincing more people of the value of a degree, but only if issues with data collection and regional disparities are ironed out, according to experts.

As part of its proposals for a reformed quality regime, the regulator plans to axe its “progression” threshold and instead assess graduate outcomes using a “basket of benchmarked indicators and contextual information”.

Three key measures will inform the rating that providers receive for “student outcomes” in the Teaching Excellence Framework: graduate-level employment data; salary information; and survey responses on whether graduates feel they are using skills learned during their degree course in their job.

Whereas previously universities had to ensure that 60 per cent of their graduates were securing a good job within 15 months of finishing their studies, this time there will be no set thresholds, with the regulator promising to take into account student characteristics, subject of study and geography.

Martin Edmondson, chief executive of the Graduate Futures Institute, said it was crucial that the regulator gets this right to avoid further undermining the sector’s credibility. “Graduate jobs are so central to the public’s perception of social permission for universities to operate,” he said.

But Edmondson cautioned that the way the different sets of statistics interact with each other might pose problems for the OfS.

Students will be deemed to have a “positive outcome” on salary if they earn above a specified threshold in any one of the three years after graduation.

Survey data measuring how much graduates rely on the skills they accrued while studying will be taken from the Graduate Outcomes survey, run by the Higher Education Statistics Agency, which captures the perspectives of graduates 15 months after they complete higher education.

“One of the interesting things will be how you tie up that data, so you’re looking at the same cohort of students, despite different lag times,” said Edmondson.

Another outstanding question concerns how the OfS will achieve its stated goal of recognising regional variation in earnings.

The latest proposals reveal that the body had considered using different salary thresholds for different geographic areas. However, it has since decided, for “simplicity and consistency”, that it will instead “take account of regional variations in earnings through benchmarking”.

Edmondson said there needed to be clarity about “what data is used to understand the regional contextualisation of salaries”.

“If we don’t get the detail of it right, there is the danger of that incentive for universities to encourage their graduates to get jobs either in the larger cities or, in particular, in London,” he said – a situation that he suggested was at odds with Andy Burnham’s stated mission to address regional inequality.

“It’s well meaning in terms of trying to give a richer picture of what graduate success looks like, but I think the devil will be in the detail,” Edmondson said of the proposals overall.

Jisc’s in-house specialist on the graduate labour market, Charlie Ball, said that earnings data was always quite a blunt tool, particularly with the working world changing so fast.

He said that taking “a snapshot of data early in graduates’ careers on salary tells us almost nothing about their likely eventual returns” but that it was very hard to calculate lifetime earnings.

Ball said we “simply don’t have the data” for those later on in their careers, and it is impossible to predict what the labour market is going to look like in 45 or 50 years.

He also noted issues with the availability of data and variance between careers – with those who are self-employed often rendered invisible by this kind of tracking.

Paul Wiltshire, who campaigns against “mass higher education” as the founder of University Watch, had a more general problem with the proposal.

He gave the example of a history graduate who went on to work as a buyer in Marks and Spencer, a position that would make very little use of their actual degree, even if it did appear to be a good job on paper.

“The biggest, most fundamental nonsense about the whole thing is that it acts on the basis that every single thing that happens in someone’s career is down to a degree – that it’s all causation,” Wiltshire said.

georgia.luckhurst@timeshighereducation.com