A quality game where cheating is allowed?

Are teaching quality assessments pointless? In the first of a two-part series, Phil Baty looks at the evidence.

The credibility of the teaching quality assessment has been shaken badly by its persistent failure to spot the problems at disaster-stricken Thames Valley University.

The university had been inspected four times since 1995. Not once did the Quality Assurance Agency assessors identify the serious problems, exposed in a special institution-wide audit, which would eventually lead the QAA, in November last year, to question the university's degree awarding powers and to send in external supervisors to secure standards.

In two subject areas - linguistics and sociology - TVU received the equivalent of excellent, with an aggregate TQA score of 22 out of 24 across six aspects of provision. Indeed, both of these subjects specifically received the top grade - four - for quality assurance and enhancement. Another subject, modern languages, gained 18 out of 24, while American studies got a rather poor 15 out of 24, although it still won QAA quality approval.

Peter Milton, director of programme review at the QAA, denies that the TVU situation exposes the failure of the system. "If you look at what happened there," he said, "you will see that the two subjects that did very well in 1995 were genuinely good - TVU got a top research assessment result in sociology. In 1994 we did pick up that their computing course was unsatisfactory, and it just scraped through the reinspection.

"We were alerted to problems in 1994, but the next two assessments happened to be in strong subjects. We had identified further problems with American studies in March 1997, but then we were overtaken by events."

It was a special institutional audit, initiated by the university's governors - not the QAA - after press exposure of problems, that eventually uncovered the deep-rooted collapse of quality control at the university.

However, a March 1995 audit by the predecessor of the QAA had raised some concerns.

Statistical anomalies further undermine the system. In 1996-98, for example, Leeds University failed its assessment in communications and media studies. Its aggregate score of 20 out of 24 included a single grade one, a failure.

But in the same subject review, in the same round, Sandwell College had its quality approved with a dramatically lower aggregate score. It got 13 out of 24, with five grade twos and one grade three.

For many critics the TVU affair and other anomalies highlight the irrelevance of the teaching quality assessment system. One critic is Roger Brown, principal of Southampton Institute and the former chief executive of the now defunct Higher Education Quality Council, which was replaced by the QAA. For Dr Brown, the TVU story shows the system is barely worth the time and money.

"It could well be the case that standards were genuinely respectable in those subjects that were assessed favourably at TVU," he said. "But as this shows, they can only ever be snapshots. What tiny fraction of what a department does is looked at during the assessment? And what proportion of the whole university's activities do the four courses represent?" Dr Milton said: "With an overall programme of assessment, spread over eight years, it is inevitable that it will take a while to pick up significant weaknesses across the board."

Dr Brown believes the latest figures for the entire round of 482 visits in 1996-98, published last month, show that the system is not working. "We should not put so much weight on something which can only ever be a very partial guide to what is happening."

Fewer failures and more passes The data from the last full assessment round have been analysed for The THES by Simeon Underwood, head of the professional courses unit at Lancaster University, which runs national courses on preparing for the TQA. The figures show grades continuing to rise, with just a tiny fraction of failures.

In the 1996-98 round, 6 per cent of the inspected courses gained the maximum possible score in all six assessed aspects of provision - 24 out of 24. This compares with 0.6 per cent of courses found to be failing.

Comparison of the two completed rounds under the current system shows an upward drift in the overall scores. The total average score for the 1996-98 round was 20.45 out of 24, compared to 20.05 for 1995-96. The final 1998-2000 round already shows even higher grades, the QAA has confirmed.

The upward drift is accompanied by an ever diminishing number of courses judged to be failing.

In the 1996-98 round, the QAA inspected 482 higher education courses in 16 subject areas. With each course receiving a grade for each of six aspects of provision, a total of 2,892 inspection grades were given. But of the 2,892, just three (0.1 per cent) grade ones were given. This compares to 1,374 (47.5 per cent) excellent grade fours.

In the first round of the current system, in 1995-96, just two (0.12 per cent) of the grades, from a total of 1,632, were failures, compared with 42 per cent of grade fours.

The proportion of failures in higher education contrasts with the further education sector, where 5 per cent in 1995-96 were deemed failing, and with schools in the same period, where one in ten secondary schools were identified as failing.

The new system has clearly decreased the proportion of failures and increased the proportion of excellents.

Before the current system, courses were marked on a simple three-grade scale - excellent, satisfactory and unsatisfactory. In 1992-95, there were 1.2 per cent "unsatisfactory" courses, compared with 26 per cent of courses judged excellent.

"Perhaps this low failure rate is not bad enough to be credible," said Mr Underwood. "Clearly it is a matter of some concern to the QAA."

The QAA's Dr Milton accepts that there are perhaps too few courses judged to be failing. "But even though there is a low proportion of grade ones," he said, "it does not mean that everything in the garden is lovely. A substantial number of courses received grade twos, which indicates significant scope for improvement."

Grade inflation?

Do the higher grades, or what critics call grade inflation, come as institutions learn to play the system? "There is clearly gamesmanship," said Dr Brown. "Academics are clever people, they can find a way round. The longer it has been going on, the more people work out how to play it. There is a whole infrastructure of training and information to help people get the best out of the system."

Indeed, Mr Underwood's unit at Lancaster University is specifically geared to helping institutions to prepare for the TQA.

"Intelligent people can learn the rules of the game," said Dr Milton. "Although it is too serious a process to call it a game. But institutions have got better at running the visits. I'm not saying there is corruption, but there is manipulation."

But Dr Milton will not concede the other main charge - that peer reviewers are too lenient when judging colleagues in their field. "There is no real evidence for that," he said.

Dr Brown disagrees. "The sector is being too lenient with itself," he said. This is exposed by the tiny proportion of failures.

Dr Milton also insists that teaching quality assessments have raised quality.

"There is no doubt that the process of making staff concentrate on the quality of teaching has improved provision," he said.

A toughened system until 2000 A 1997 research report, Improving the Quality of Education; the Impact of Quality Assessment on Institutions, found that two thirds of the quality improvement recommendations made by inspectors had prompted institutions into action.

Under increasing pressure to provide tangible, quantifiable evidence that public money is being well spent in higher education the QAA has moved the goal posts again to try to catch and identify more problems, and to insist that warnings are being acted on.

In a letter to vice-chancellors in November last year, Dr Milton, said: "The QAA and HEFCE now judge that additional follow-up for those subject providers which, although approved, show several areas of weakness in their graded profiles is likely to be helpful in terms of working with institutions to secure quality enhancement.

"The agency and the HEFCE have concluded that there should be follow-up action in the small number of cases where a graded profile contains three or more grade twos," he said.

"Such a profile indicates significant room for improvement. In these cases, the agency will seek from the subject provider a plan for improvement setting out the remedial action to be taken."

A re-visit will occur. All institutions inspected after October 1998 would be subject to this tougher system.

The new catch-more system would have caught at least one - American studies - of the four courses given the all-clear at TVU, but it would have left the other three subject areas coming out clean.

"Would the procedure the agency is introducing in the quality assessment exercise have been any help in identifying the major problems at TVU the audit has identified?" asked Mr Underwood.

Indeed, the toughened system would have meant re-visits to 16 institutions from the whole 1996-98 round, not just to the three which failed. But 16 is still just 3.3 per cent of institutions on top of the 0.6 per cent which failed.

According to Mr Underwood, there could still be "striking statistical freaks" under the new tougher system. Take the TQA grade for provision in French at the British Institute in Paris. "It achieved an aggregate score of 18, comprising three grade fours, and three grade twos," he said. "Under the pre-1995 methodology it might have been given an 'excellent' grading; under the new 1998 procedure it would have had to go through the trouble and uncertainty of submitting a plan and preparing for a re-visit."

For Mr Underwood, the new system poses as many questions as it answers. "Does the subject provider have a right of appeal?" he said. "What happens if the agency doesn't like the improvement plan it submits? Who undertakes the visit one year on? And, most importantly, what happens if on the re-visit the agency decides that not enough progress is being made in securing necessary improvements?

"This is the bottom line; under present procedures, a unit which on re-inspection is still unsatisfactory has its HEFCE funding withdrawn; would that happen here?" Apparently not. The re-visit resulting from any course receiving three or more grade twos, will have little status. There will be no formal re-assessment, the QAA has confirmed, as in cases of failure. Just one representative from the agency will revisit the institutions within a year, "in order to discuss progress".

Perhaps these are the reasons why the TQA will be replaced in 2001. "The system has outlived its mode," said Dr Milton. But he adds that for all its faults the system has boosted the profile of teaching. And many critics, including Mr Underwood, would agree.

All data supplied by Professional Courses Unit at Lancaster University.

How they fared in the QAA game

Top departments (gaining 24/24) in the 1996-98subject review round Americans studies at Keele, Central Lancashire, East Anglia.

Building at Kingston.

Communication and media studies at Chichester Institute of Higher Education.

Drama, dance and cinematics at Lancaster, Hull, Kent, Reading, Warwick.

Electrical engineering at Imperial, Queen's Belfast, Birmingham, Bristol, Essex, Huddersfield, Hull, Sheffield, Southampton, York.

General engineering at Open University.

History of art at Birkbeck, SOAS, UCL.

Materials technology atImperial.

Mechanical engineering at Kingston, Nottingham.

Town and country planning at Oxford Brookes, Greenwich.

Failed departments in the 1996-98 round Communication and media studies/dance, drama and cinematics at Bolton Institute.

Communication and media studies at Leeds.

Building/civil engineering/electrical engineering at Stockport College.

Departments which would have failed under the tighter QAA procedures brought in last autumn Agriculture at De Montfort.

American studies atThames Valley.

Building at North East Surrey College of Technology,Wigan and Leigh College.

Communication and media studies at Sandwell College,St Helens College, SuffolkCollege, East London,Wirral College.

Dance, drama and cinematics at City College, Manchester, Edge Hill, Hertfordshire.

Electrical engineering at the City of Liverpool Community College, Central Lancashire, East London.

General Engineering atRycotewood College.