Encounters of an unfair kind

Why do small subject areas and old universities do better in teaching quality assessments? In the second part of our analysis, Phil Baty questions peer review

Academics acting as teaching quality assessors may be more lenient towards academics they know than those they do not know, according to an analysis of teaching quality assessments. The analysis will fuel concerns that peer review cannot safeguard standards in universities.

As The THES showed last week, the full results of the 1996-98 round of teaching quality subject reviews revealed a continuous rise in grades and just a tiny, shrinking proportion of failures. The Quality Assurance Agency said that the figures showed that the exercise had naturally led to an improvement in the quality of teaching, but the agency also accepted that gamesmanship was partly to blame as institutions learned how to get the best out of the system.

The QAA insisted that there is no hard evidence that the sector is being too easy on itself. But Roger Brown, former chief executive of the now defunct Higher Education Quality Council and principal of Southampton Institute, believes the peer reviewers are finding it difficult to dish out damning marks to colleagues.

Small subjects do better "The QAA's reviewers are not inspectors, they are academic peers and the sector is being too lenient with itself," said Dr Brown. "In the smaller groupings you appear to have something close to collusion between the assessors and the assessed."

The data appear to back up his concerns. A pattern emerges when the size of a subject group is measured against its performance in the TQAs. Smaller groups - where there is a more intimate relationship between reviewers and reviewees - tend to do better.

Of the 16 subjects assessed in the 1996-98 round, by far the highest scoring subject area was East and South Asian studies. Almost 90 per cent of departments in the subject group were awarded between 21 and 24 out of 24. The East and South Asian studies group consists of just nine departments and the work was assessed by ten peer reviewers drawn mainly from the universities assessed.

American studies, another high-scoring area, with 75 per cent scoring between 21-24, was made up of just 20 departments. There were 33 subject specialist assessors. Another high scorer was Middle Eastern and African studies, with nine departments, and just seven assessors, where 67.7 per cent of those assessed achieved between 21-24.

The lowest scores cluster in the largest subject areas. In communications and media studies, where 60 departments were assessed, just 35 per cent scored more than 21. Of the 71 inspected in electrical engineering, 46.5 per cent scored more than 21.

There are exceptions. Food science was one of the smallest groups, with only 11 departments, and only two departments scored more than 21 out of 24. But of the 16 subjects assessed in the 1996-98 round, 56.25 per cent of departments were deemed excellent in the smallest eight subjects, compared with 50 per cent excellents among the largest eight.

Peter Milton, director of QAA programme review, said it was aware that small subject areas often did better than larger groupings and that it was "undoubtedly a matter of concern".

"But it is not surprising in a peer review system. Some smaller subject areas have had higher grades overall, but some have also fared less well."

Dr Milton said that the picture may also be skewed by the type of institution assessed. The best scoring subject areas tend to include a high number of the more prestigious, old universities, which almost invariably score better, he said.

Indeed, the highest scoring subject group, East and South Asian studies, is not just the smallest group, it also consists almost exclusively of old universities, with just one former polytechnic. The worst scoring subject area, food science, is made up largely of former polytechnics.

Old better than new The clear difference between the old and new universities, said Dr Brown, is a further sign of system failure. The assessments are explicitly graded against the institution's own stated aims and objectives.

The TQA reports contain the premise that higher education institutions "vary greatly in size, subject provision, history and statement of purpose. Each has autonomy to determine its institutional mission and its specific aims and objectives at subject level. Quality assessment is carried out in relation to the subject aims set by each provider".

But from the data it would seem that old universities are better than new ones at meeting their own objectives and new universities are better than colleges at doing so.

Old universities score an average of 21.5 out of 24. Former polytechnics achieve an average of 20 out of 24, while the college sector languishes behind with 18.9. The college differential is exacerbated with an average 18.48 score if the usually high-scoring monotechnics and specialist institutions are removed.

Indeed, of the three departments that failed the 1996-98 round just one was an old university, Leeds, which failed, the QAA says, on a technical administrative point related to media studies provision, not on the quality of teaching. Leeds also passed its reinspection. Of the 16 courses that would have fallen foul of the new tougher system being introduced - under which an institution is reinspected if its course scores three low grade twos from the six aspects of provision - not a single course was provided in an old university. Six of the 16 were new universities and the rest were colleges of higher education.

The only old university to gain three low grade twos since the system began in 1995 is the University of East Anglia, for sociology. UEA was one of the first to be assessed under the correct methodology.

The colleges and new universities at the bottom end of the scale contrast sharply with the 29 courses given the top mark of 24 out of 24 in 1996-98. Just six of the 29 are former polytechnics and Chichester Institute of Higher Education gained full marks.

"The TQA is supposed to be based on institutions' own stated aims," said Dr Brown, "but can the QAA really say that old institutions are almost always better able to achieve their objectives than new universities? Or is it simply bias on the part of the assessors?" Simeon Underwood, who runs the professional courses unit at Lancaster University, and who compiled the TQA data for The THES, said: "It simply should not be like this. The evidence that assessors are going into institutions with preconceptions is unarguable."

The QAA's Dr Milton makes no bones about it - old universities do better because they are better resourced. "The former UFC institutions have better research ratings on average and hence better resourcing. Because we look at a range of issues related to quality, these better resourced institutions are always likely to do better in the overall gradings."

Money is clearly an issue - college courses tend to do badly compared with universities in all six aspects of provision, but they are weakest in the learning resources category, which is a largely cash-based judgement, examining library and laboratory facilities.

Mr Underwood cites material from the QAA's own former guidebook on the subject reviews that says "assessors should be aware that very good teaching and learning can take place in unsuitable conditions".

"The exercise should not just be about the quantum of resources and the wealth of institutions. If it is, why bother?" said Mr Underwood. "Why not just measure institutional wealth? Do the figures really mean that the older universities are so significantly better at this?" An analysis of the results in the old university sector suggests greater complexity. The younger old institutions of the 1960s tend to outperform the large ex-civics that have "wonderful resources", said Mr Underwood.

"The Warwicks, Yorks and Lancasters are outperforming the Manchesters, Leeds and Birminghams."

He also points out that new universities and colleges do not just lag behind in the assessment of learning resources.

"The pattern of the hierarchy is absolutely clear in all the aspects of provision with the single exception of quality management. In all of the others there is a very clear gap between ex-university funding council institutions and the rest," Mr Underwood said.

Variations in grading Another anomaly that emerges from Mr Underwood's analysis is the variation in the grades given in each of the six aspects of provision. It is clearly easier to be judged excellent in some aspects than others.

In student support and guidance, 77 per cent of the grades awarded were the maximum grade fours. It is harder to get excellents in teaching, learning and assessment, where only 26 per cent are grade fours, but where threes are extremely common. It is easiest to slip up in quality assurance and enhancement, where there is a high proportion (17 per cent) of grade twos, compared with 7 per cent twos in learning resources and 5 per cent in teaching learning and assessment.

Dr Milton is aware of the problem. "It is true that the grades for student support and guidance are higher than other aspects," he said. "There are many reasons for this but the main one is that universities and colleges are generally pretty good at looking after their students."

But some believe that the category can be a "soft option" for the assessors - a few too many excellents in one aspect of provision could compensate for tougher judgements about another.

Analysts such as Mr Underwood believe the breakdown between the aspects reveals something of the thinking peer assessors take into the exercise. "The lowest proportion of grade fours is given in the teaching, learning and assessment category," he said. "But there is also a very low proportion of grade twos in that aspect. This could mean that peer reviewers are very reluctant to give poor grades to colleagues in an area which relates directly to their performance."

The argument also runs with the quality assurance and enhancement aspect relative to other aspects. "There is a very high proportion of twos in this aspect relative to other aspects," Mr Underwood said. "It could be that this aspect is used to calibrate the overall score. If an assessment team has a preconceived idea about an aggregate score for a certain department, the quality aspect could be used to lose the odd point."

Dr Milton will not accept such criticisms, but he accepts that the system needs an overhaul. He is devising details of a radical approach to quality assessment to replace the TQA regime when it comes to an end in 2001.

"The TQA has been an essential pre-requisite for the move towards a more flexible system," he said. "TQA has provided some of the essential ingredients for rapid moves towards a lighter touch for institutions which can demonstrate their high quality and high standards." There will be a tougher approach to those that cannot.

But the system, Dr Milton accepts, is rapidly losing the sector's confidence. "At the end of its first cycle (in 2001), this subject review method will clearly have had its day," he said.

WHAT THE REVIEWERS ARE LOOKING FOR

Each course is assessed in six aspects of provision:

* Curriculum design, content and organisation

* Teaching, learning and assessment

* Student progression and achievement

* Student support and guidance

* Learning resources

* Quality assurance and enhancement.

Each of the six aspects is given a grade from one to four. A single grade one in any of aspects constitutes failure, with "major shortcomings that must be rectified".

Grade two signifies that the aspect of provision "makes an acceptable contribution to the attainment of the stated objectives, but significant improvement could be made".

A subject can win quality approval as long as it does not receive any grade ones, but under new rules, if a subject receives three or more grade threes, it will be subject to an informal follow-up meeting with a QAA assessor, to ensure it initiates improvements.

Grade three is usually considered good, with the aspect "making a substantial contribution to the attainment of the stated objectives", but with room for some improvement.

Grade four is the best, with the aspect making "a full contribution to the attainment of the stated objectives".

All data supplied by professional courses unit, Lancaster University.