Phil Baty looks into claims that the Quality Assurance Agency is unaccountable and may not be providing the university sector with value for money.
The question "who quality-assures the Quality Assurance Agency?" is one that Geoffrey Alderman, pro vice-chancellor responsible for quality and standards at Middlesex University, wants answered.
Professor Alderman, a council member of the Society for Research into Higher Education, is one of the QAA's most vociferous critics. He believes the QAA should be subject to proper third-party scrutiny.
"In every other part of the quality assurance industry those who quality assure are themselves quality assured - usually by the United Kingdom Accreditation Service," he said. "The QAA is deficient in this respect. It needs to be accredited and it needs to be done very soon, as a lot of us in the sector have serious worries. If there was a third party, it would give some confidence."
The QAA insists that it has sufficient third-party scrutiny in its relationship with the Higher Education Funding Council for England, which audits the QAA, and through its legal status. "The agency is subject to the usual accountabilities of a company and a charity," said a spokesman. But critics believe it should be more accountable to universities.
Individual universities fund the agency to the tune of Pounds 3.2 million each year in compulsory subscription fees, which make up 38 per cent of its total income. Almost all of the rest of the QAA's annual income of more than Pounds 8 million comes from the university funding councils in England and Wales (which pay the QAA to finish their cycle of teaching quality assessments) and from the Scottish funding council and Northern Ireland Office, which will join an integrated quality regime next year. These pay Pounds 4.2 million through contractual service level agreements. In 1997 - the most recent available financial information - the QAA generated just Pounds 300,000 of its own income.
Both the Higher Education Funding Council for England and the QAA declined to show The THES a copy of the service level agreements, which spell out how the QAA is accountable for the Pounds 8 million of public money it spends annually.
A QAA spokesman said that the "detailed terms are a matter for the contracting parties". The agreements focus on the funding councils' legal obligations to ensure that an effective system for monitoring teaching quality in higher education is in place and it allows the councils to audit the QAA.
The QAA was set up by the sector, for the sector, in 1997, to "provide an integrated quality assurance service for higher education institutions throughout the UK", it says. Merging the quality assessment functions of the English and Welsh funding councils and the institutional audit work of the Higher Education Quality Council, the QAA is now working on a fundamental new quality assessment methodology.
This has yet to be agreed by the funding councils and has yet to be fully approved by the sector at large. Scheduled to be published next month and piloted for implementation next year, it will be the true measure of the QAA's success. But many in the sector believe the agency has got off to the worst possible start.
The question of "ownership" is Professor Alderman's most rudimentary concern.
Technically the QAA is "owned" by universities. A company limited by guarantee, its members are the Committee of Vice-Chancellors and Principals, the Standing Conference of Principals, the Heads of Higher Education Wales and the Conference of Scottish Higher Education Principals.
The exact status of the representative bodies is unclear and the QAA has declined to discuss the exact nature of the accountability mechanisms or declined to provide a copy of articles and memoranda that spell out the relationship.
Despite these formal legal mechanisms, there is a problem with communications with individual institutions and academics. "There seems to be no accountability back to the sector which funds the QAA," said Professor Alderman. "Usually, he who pays the piper calls the tunes. But with the QAA, the sector pays the piper but we have no way of calling the tunes."
Consultation.
In November 1998, the QAA wrote to vice-chancellors and principals with a "proposal" to beef up checks on teaching quality standards. After explaining the proposals to change the criteria for subject review follow-up visits, the QAA's circular went on to explain that the "proposal" had already come into effect a month earlier.
The QAA's plans to change the law to allow the removal of failing universities' degree-awarding powers were never shared with the sector at large and were put to ministers earlier this year despite vociferous objections from both the CVCP, which said the plans showed the QAA's "profound mistrust of the whole university sector", and from SCOP, which said the plans threatened "the vitality, diversity and future development of higher education colleges".
The QAA said the degree-awarding powers' plans did not fall under the usual consultative processes, as ministers had specifically sought direct advice.
"The review of the criteria was undertaken for government, which wished to, and did, conduct its own consultation," said a spokesman.
There has also been controversy about its use of working groups to help it develop policy. When the QAA called off the final meeting of its working group on student complaints last week, one member, Don Staniford of the National Postgraduate Committee, said that it had been used as "mere window dressing and a blatant attempt to muzzle criticism".
The Standing Conference of Principals - one of the QAA's four "members" - insists it is generally happy with the situation. "It's more about partnership than control," said Greg Wade, SCOP's administrative officer.
But he added: "At the moment we have supported the QAA. But we will be asking if we can send in an observer to the board. This will enable us to have even closer links with the QAA."
The CVCP declined to comment on relations with the agency.
Value for money.
Of the Pounds 8.2 million the QAA spent in 1997-98 a quarter went on administration. About Pounds 4 million was spent on teaching quality assessments (159 reports in 1997-98), an activity that will continue for another year.
Just over Pounds 1 million was spent in 1997-98 on 33 institutional audit reports. One of the key areas of the agency's remit is to "promote public confidence that quality of provision and standards of awards in higher education are being safeguarded and enhanced". But some critics fear the audit process has become meaningless.
Simeon Underwood, head of the professional courses unit at Lancaster University, said: "The reports that are produced on the audit visits are long, hard to read and difficult to gauge. It is possible for two different people to read the same report and come to diametrically different views about whether (the QAA) likes what the institution is doing or not. They are written for a narrow readership, and are of little use either within individual institutions or more widely. It would be nice to know that the QAA had thought long and hard about the cost and benefits of this exercise as carried out at present."
One audit report, on the University of Bradford, Mr Underwood notes, appears to contradict itself: "It is not easy to confirm that an effective system exists ... to enable the institution to satisfy itself about the quality of its educational provision," said one part of the conclusion. And another: "The university is exercising adequate stewardship over its academic quality and standards."
Indeed, one mischievous THES reader put the text of the QAA's audit report on Essex University through a software programme designed to check grammar and "readability". The programme gave the document a "Flesch reading ease" rate of 16.2 out of a maximum of 100.
The QAA declined to comment on complaints that the reports were "barely comprehensible" without "further particulars of the alleged criticisms".
A particular irritant for some is the fact that the information which would allow the sector to make a judgement about the QAA's activities is sketchy.
Mr Underwood said: "Is it providing value for money? It is difficult to tell from the outside, and the budget summary on the QAA website does not provide enough detail to tell. It would be nice to know that the QAA is using internal indicators such as the average cost of each subject review, and was prepared to share these figures with the sector. It would also be nice to know that the QAA is consciously seeking to economise not just internally, but in its dealing with institutions."
Mr Underwood fears, like some of his colleagues, that the QAA's new methodology may fail to achieve its primary aim: to reduce bureaucracy and cost.
He said: "One of the reasons the agency was set up was to streamline the burden arising from the double system of audit and assessment, but in my experience most institutions don't believe that this has happened. Also, there is already strong doubt in the sector that the new Framework after 2000 will reduce the burden, and the QAA has yet to provide convincing evidence that this is even one of their priorities in constructing it."
This week, the QAA said: "The agency was established to ensure that there can be public confidence that quality of provision, and standards of awards in higher education are being safeguarded and enhanced. Integration of this process will come with the introduction of the new method of quality assurance."
Oliver James, lecturer in public policy at Exeter University, has just finished a study with the London School of Economics on the growth of regulatory bodies.
He said: "The big question for the QAA is whether they have done any work on the costs to universities of compliance with their requirements. From my personal experience we are investing quite heavily in putting quality assurance systems in place, and it's a large extra burden. In the private sector the costs of compliance are always calculated."
Standards in public life.
In October 1998, a year and two months after its birth, the QAA said that its 13-strong board was "developing a Code of Best Practice that will allow it to meet the standards of best practice in public life as articulated in the Nolan reports and the principles and practice of effective corporate governance as articulated in the Cadbury, Greenbury and Hampel reports."
However, these three reports were directed at businesses rather than non profit organisations.
In May this year, the QAA revealed that it had been audited by HEFCE under the terms of its service level agreement. Although neither HEFCE nor the QAA will publish the audit report, 22 recommendations for improvements in management and governance were made.
One of the key points raised by HEFCE auditors was poor strategic planning by the agency. The board has confirmed that it has now adopted a strategic planning model and has written an "action plan" to address all the criticisms raised by HEFCE.
But there is a general uncertainty about how the QAA makes decisions over appointments and remuneration.
Last year, the QAA's own staff were so sure that John Randall would promote his former Law Society colleague Julie Swan as his director of development that they announced her appointment in a spoof press release before the agency had finished interviewing for the post. A week after the mock press release announced her appointment, Ms Swan was confirmed in post, becoming a director at the QAA after just one year, and joining another former Law Society colleague Stewart Bushell.
The QAA insisted at the time that Ms Swan was appointed on her merits, and said that she had been interviewed by a panel of two people, including Mr Randall.
The spokeswoman confirmed that while Mr Randall had full personal discretion over appointments at the QAA, he had initiated a policy to ensure that appointment panels were made up of at least one other person.
The QAA said this week that it now conforms to the Nolan principles regarding decisions over remuneration and appointments.
Alternatives
Exeter University's Mr James has some sympathy with Professor Alderman's suggestion that the QAA needs more third-party scrutiny. "Our regulators tend not to be subject to the same standards of validation as they subject others to. They are very open to accusations of double standards."
But it could be problematic. "There is a real question about who guards the guards but it could be infinitely regressive and could be absurd," he said. He has a more radical idea - a competitive quality assurance regime.
"When you have competition between regulators for clients - with universities choosing how and under which system they are accredited - the regimes will be more responsive. It happens with auditors in the private sector. The down side of a competitive system is that it could cause confusion among students who will not know which particular standards are adhered to by which institution.
But there is an extent to which single national regulatory systems can lead to an averaging out of standards and that stifles innovation."