Analysis: Worthy project or just a game?

The TQA cost more than £100m and lasted ten years. Phil Baty draws on exclusive research compiled by The THES and asks: was it worth it?

The ten-year project to judge the quality of teaching in every United Kingdom university department has not been worth the huge expense, or the incalculable cost to staff morale. An analysis of the teaching quality assessment exercise by The THES underpins the long-held views of academics and quality assurance specialists that the system was flawed.

The analysis covers more than 8,300 individual grades awarded in reports on more than 1,300 university departments published since the exercise began in its numerical form in 1995.

It provides strong statistical evidence that many academics' worst fears may be justified: the project has been blighted by elitism, favouritism, gamesmanship and grade inflation. Moreover, it seems that despite attempts to factor it out of the system, money remains a determining force in a department's assessment results.

The Quality Assurance Agency has refused to talk to The THES about the analysis, but ministers have backed its insistence that the exercise has been a resounding success. As education secretary David Blunkett said last week: "We have published a great deal of valuable information, and I pay tribute to the work that has been done by the QAA. It has rightly attracted international respect."

Even the exercise's worst critics concede it has brought some benefits to higher education. Geoffrey Alderman, a former TQA assessor who runs courses on how to maximise TQA scores, said it had taken dud teachers off the job, helped universities to find money to improve libraries and even spruced up lecture rooms with a new coat of paint.

But he said: "Of course, this could have been done without the paraphernalia and expense of the TQA. The real expense has been enormous - incalculable - and stress levels in higher education institutions have risen as a result."

The TQA, or subject review as it is officially styled, has cost every department in every university from £20,000 to £200,000 in paperwork and staff time, for each of the QAA's 1,300 visits. The total cost to the sector is approaching £100 million, plus another £3 million to £5 million a year in official administration costs incurred by the QAA drawn from the public purse. Altogether it is a huge amount of money for an exercise many critics claim is unsound.

GRADE INFLATION?

• The proportion of departments
  achieving 22 or 24 (excellent) has
  increased from about 25 per cent
  in 1995-96 to 60.5 per cent in
  1998-2000.

One allegation made persistently by the TQA's critics is that the exercise is being rendered all but meaningless by grade inflation.

The THES 's data are unequivocal: university departments' TQA results keep getting better, and are set to jump further in this year's final round, according to preliminary indications, which show more than 70 per cent excellence.

The average scores are also increasing: from 20.04 out of 24 in 1995-96, to 20.40 in 1996-98 and up to 21.70 in 1998-2000. Of course, this could suggest that provision is improving, not least as a result of the TQA. But, for some academics, the figures are too dramatic to be explained in such terms.

Roger Cook, Napier University's academic development adviser, compiled the data on behalf of The THES . He is convinced that gamesmanship is coming into play. "I firmly believe that this is a product of institutional learning," he said.

The exercise judges institutions against their own stated aims and objectives, and Mr Cook believes institutions have become adept at writing what the inspectors want to see. Academics have also been taught how to impress the QAA through training industry courses where assessors themselves often reveal the secrets of the process.

As Professor Alderman puts it: "The TQA has given rise to a quality assurance industry, of which I am a part. It's become outdoor relief for underpaid academics."

QAA chief executive John Randall recently attacked as cynics those who say the results reflect the increasing invalidity of the subject reviews.

PROPORTIONALITY?

• In 1998-2000 just 0.1 per cent
  of the 3,990 inspection grades
  given were fail grades.

The flip side of the grade inflation debate is that only a tiny proportion of departments are ever judged to be failing.

In the whole of the 1996-98 round, just three departments were found to be failing. From a total of 2,892 individual grades dished out in the round, just 0.1 per cent were fail grades (grade one out of a possible four).

In the last full round, from 1998-2000, four departments failed: Derby University failed a joint inspection of its medicine and pharmacy review, Liverpool University failed its nursing review after a technical hitch, and two colleges, Wigan and Leigh, and St Helens, failed inspections of their art courses. These four failures represent only 0.6 per cent of the 665 reviews. This is what has convinced most academics that the exercise is a waste of resources.

The Association of University Teachers this week called for a fundamental review of the QAA system. "Quality assurance is vital, but the system used to secure it should be in proportion to any problems," the AUT's election campaign manifesto says.

CAN MONEY BUY RESULTS?

• Since the exercise began, old
  universities have achieved an
  average score of 21.55 out of 24,
  compared with 20.49 for new
  universities and 19.75 for colleges.

Critics have also claimed that the exercise has been too sensitive to money. The THES 's figures reveal a hierarchy, with the richer old universities at the top of the tree. "The difference between the type of university seems to point to a story that resource base underpins performance," said Mr Cook.

The QAA asserts that departments should be judged against their own stated aims and objectives. Its guidelines for reviewers state clearly: "Assessors should be aware that very good teaching and learning can take place in unsuitable conditions."

But the money factor becomes even more notable on examination of the different aspects of provision that the QAA looks at.

The data support the idea that money helps. In the categories of inspection most closely linked to an institution's financial position - learning resources, which includes judgements on library and computer facilities - richer universities win. Old universities achieve an average of 3.85 out of four for learning resources, compared with 3.38 among colleges.

SUBJECT VARIATIONS

• In 1998-2000, 100 per cent of
  veterinary medicine courses were
  excellent. In nursing only 54 per
  cent of courses were excellent.

This hierarchy of results has also led to criticisms that the QAA's reviews exhibit bias.

The Association of Colleges has claimed that its sector is unfairly penalised by the TQA because the QAA's review teams are dominated by staff from the old universities, who carry preconceptions. The QAA appeared to accept this when it launched a drive to recruit more college-based reviewers.

The THES 's analysis shows significant variations in the performance of institutions between subject areas. Although there are exceptions, smaller subject groups perform better overall than larger areas, suggesting leniency among close colleagues.

Perhaps the most extreme example of this is in veterinary medicine. With only 12 QAA-appointed reviewers and four departments inspected, three achieved the full 24 out of 24, and the fourth, at Cambridge, achieved 23 out of 24.

Similarly in dentistry, where 12 departments were assessed by 26 specialist reviewers, some 75 per per cent of departments received excellent grades of 22 or more out of 24. In pharmacology and pharmacy, where 26 departments were inspected, 77 per cent of departments achieved 22 or more.

In contrast, a below-average 54 per cent of departments achieved excellent ratings in nursing, where 66 departments were inspected. And only 54 per cent achieved 22 or more in mathematics, where 71 departments were inspected.

THE FUTURE

While the TQA system is being brought to a conclusion, with an 11th-hour £5 million dash to finish the ten-year job by the end of 2001, the QAA is already rolling out its new and supposedly improved system of subject review in Scotland. It will launch the system in England in January next year.

The agency is promising that its framework will cut the burden of scrutiny dramatically, a promise that was bolstered last week with the intervention of Mr Blunkett, who committed the QAA to a 40 per cent cut in teaching quality assessment visits in the future.

The basis of Mr Blunkett's announcement that a much lighter touch can be introduced in the future was his assertion that the high level of attention given to universities' teaching quality in the TQA justifies placing a greater reliance on institutions' own internal quality assurance.

Few in the sector will question the need for a lighter touch, but many may doubt the usefulness of the thousands of reports compiled on quality over the past decade. "The dons have outsmarted the government by turning the exercise into a game and playing it brilliantly," Professor Alderman said.

More details in the Statistics section
» Teaching Quality Assessment trends 1995-2001