Do universities teach students to think critically?

Professional services giant PwC’s recent announcement that new recruits will no longer require at least a 2:1 degree was seen by many as the latest sign that some of the world’s largest employers are losing faith that a good university qualification guarantees a candidate of a certain quality.

The firm is by no means the first to look for new ways of determining the talent and potential of recent graduates as employers become increasingly vocal about the supposed failures of even the top universities to ensure that those entering the workforce have obtained the status of being “job-ready”.

In response, governments and policymakers around the world have emphasised the need for more practical, vocational degree courses that are closely tied to real-world experiences. But a new publication from the Organisation for Economic Cooperation and Development (OECD) argues that it is in the teaching of more generic critical thinking skills where universities can make the most difference.

“There’s a discrepancy in that people are qualified – they have the stamp from universities that says they can do certain occupations – but then employers find they don’t have the skills needed for the workplace,” said Dirk Van Damme, who co-edited the new book and recently retired as the OECD’s head of innovation.

“The assessment done by universities doesn’t guarantee that candidates have the problem-solving skills that employers think are important, and so they have to find ways to test this themselves.”

The notion that institutions are lacking in this regard has long been suspected, and the researchers behind the study think they may finally have come up with a way to prove it.

If all this sounds familiar, it is because it is. Many of those involved in the research also worked on the OECD’s aborted Assessment of Higher Education Learning Outcomes (Ahelo) project, which sought to establish a global system for assessing students’ skills at the end of their degrees.

Billed as a university-level equivalent of the highly influential Programme for International Student Assessment (Pisa) tests for school pupils, the scheme faced stiff opposition from elite institutions – some of which were arguably motivated by fears for their positions at the top of the hierarchy if teaching outcomes were to become better known.

More fundamental questions were also raised about whether such skills could accurately be assessed across institutions and borders, and the project fell apart in 2015.

A handful of countries remained committed to the idea, however, and have been testing students’ critical thinking skills ever since using the Collegiate Learning Assessment (CLA+), developed by the Council for Aid to Education (CAE), a US-based non-profit. The assessment includes a performance task and set of questions designed to test a student’s cognitive thinking, rather than their ability to recall knowledge.

“There’s no way that any one specific assessment can measure all of critical thinking,” acknowledged Doris Zahner, CAE’s chief academic officer and the co-editor of the new book.

“What we do really well is measure a specific, well-defined component of critical thinking: namely, analytical reasoning and evaluation and problem-solving,” she said.

“That includes data literacy, understanding quantitative information, being able to gather information from various sources and then making a decision based on this and crafting an answer that supports your argument and refutes the opposite – that’s what the assessment does.”

The results of the tests, published by the OECD on 30 August in the book Does Higher Education Teach Students to Think Critically?, are stark: on average, only 45 per cent of tested university students were proficient in critical thinking, while one in five demonstrated only “emerging” talent in this area.

What’s more, the “learning gain” of students between the start and the end of their courses was found to be small on average, while there were big discrepancies between courses, with those studying fields closely aligned to real-world occupations – such as business, agriculture and health – scoring the worst.

For Dr Van Damme, the results reflect a move away from the teaching of critical thinking in higher education, with less emphasis being placed on engaging with content and with some sectors abandoning exercises such as essay writing.

“Critical thinking is a skill that I think [many people] just assume is taught,” Dr Zahner said. But she pointed out that it has never been reported in university transcripts, so there has traditionally been no way of knowing if a student has developed these skills. “Universities, at least the ones that we have talked to, have said ‘It is not our job; they should have learned these things in high school’…everyone feels like it is somebody else’s responsibility to teach these things,” she said.

The authors recognise the limitations of the research, particularly the self-selecting sample of students, confined mostly to campuses in the US, with only a fraction coming from the other five countries taking part – Chile, Finland, Italy, Mexico and the UK – meaning that data for these countries could not be said to be representative.

But the authors believe they have demonstrated that “an international, cross-cultural, comparative assessment of generic learning outcomes of higher education is feasible”.

While the OECD does not yet seem to have mustered the will for another go at instigating an Ahelo-type project, the study’s repercussions could be major.

“What I personally believe this will do is lay the foundations for placing greater weight on the quality of teaching in higher education,” Andreas Schleicher, the OECD’s director for education and skills, told a launch event for the book in Hamburg.

He said that employers had “seen through” the degree system and that students were becoming more discerning consumers because they were having to shoulder more of the cost of their education.

Therefore, he continued, it was getting harder to “hide poor teaching behind great research”, while demand for skills that were easiest to test and teach – such as memorising and regurgitating knowledge – were exactly the areas that were losing value most quickly.

“Teaching excellence needs to obtain the same status – the same recognition – as academic research, which is still the dominant metric for valuing academic institutions, whether you look at rankings, research assessment frameworks or performance-based funding,” Mr Schleicher said.

To critics, all this sounds suspiciously like groundwork for the creation of a new ranking, something that was never an aim of Ahelo even though many thought its data were likely to eventually feed into institutions’ scores in global league tables.

Dr Van Damme said that while many criticisms of rankings were justified, there should be a recognition that they are not going to go away and, therefore, it would be better to find ways to ensure that they accurately reflect the quality of teaching – something that could change the complexion of league tables completely.

“In an ideal world, where you have as much transparency for teaching and learning as you have for research, there would be a profound impact not only on rankings but the hierarchy and landscape of the system,” he said.

“It is certainly not the case that universities that are excellent in research are also automatically excellent in teaching and learning; and if you placed greater weight on teaching, you would get different results [in rankings].”

As well as an upheaval in institutional reputation, greater focus on the teaching of critical thinking could fundamentally alter the types of courses that are seen as necessary for societies and economies to thrive, according to Dr Van Damme.

Politically influenced drives towards utilitarian approaches to education that produce students who are immediately employable in a certain occupation – which tend to favour the STEM subjects – neglect to consider volatility in the labour market and the need to train young people for their entire lifetime, he said.

“The economy and labour market are in transformation because of digitalisation, and so the job reality in 10 years’ time will be completely different from today. There should be more interest in teaching the generic skills that matter in the long term,” he added.

In this world, it is the much-maligned humanities that truly come into their own, and the CLA+ results showed that those students pursuing these fields displayed much higher levels of critical thinking, according to Dr Van Damme.

He said studies have demonstrated that while vocational training produces better employability results in the short term, these wane after five years and “those with better generic skills have much better employability and earning prospects over a lifetime”.

Dr Zahner said universities would likely come under increased pressure from industry and governments to address these issues, whether they like it or not.

“Hopefully the universities will hear this messaging. It’s great if you can graduate your students, but it is not so great if you graduate all these students and they don’t have success in their careers. We’re hoping being able to increase critical thinking skills will be able to close that gap.”

tom.williams@timeshighereducation.com