Student survey plans fall apart

The government looks set to abandon the first of its flagship student satisfaction surveys later this year amid fears that university and academic reputations will be damaged by distorted polls.

Plans for the national opinion survey, announced in January's higher education white paper, are in disarray following the disclosure that student response rates could be so small as to be, at best, meaningless and, at worst, harmful. The process could cost each university up to £155,000 a year.

Members of the steering group finalising the survey plans are alarmed that under the Australian student feedback system, which is being used as a model for Britain, judgements on the quality of subjects at some universities have been made by as few as one or two students.

Concerns that Britain's version should provide meaningful information and not damn institutions on the basis of minority opinions mean the government is almost certain to abandon its white paper commitment to publish the first annual survey in the autumn.

John Brennan, a member of the steering group and director of the Open University's Centre for Higher Education Research, said: "Clearly there is an issue about response rates with a survey of this kind, and a whole set of decisions about the timing of the survey and the mechanisms for contacting students have to be looked at in trying to ensure rigour."

In its white paper, the government promised high-quality information for students from the first national survey to explicitly cover teaching quality.

Professor Brennan said: "Any survey that is done this year is likely to have pilot status." He said that this meant the results would remain confidential.

Another steering group member said: "I think there is a bit of an understanding gap between the government and the steering group on when it will be up and running."

A spokeswoman for the Department for Education and Skills said: "We are committed to ensuring that methodology and findings for the national annual survey are reliable and meaningful. We are working closely with the Higher Education Funding Council for England to ensure that we deliver this to the timescales laid out in the white paper."

The steering group is working on final details of the plan with a view to reporting to Hefce in the next few months.

It is understood that the group has agreed on a range of survey questions, inviting students to rate their level of satisfaction on a scale of one to five on a number of elements of their course. Questions will be asked, for example, on whether lecturers were sufficiently challenging and on the quality of information and library facilities.

Education secretary Charles Clarke has said that the survey will give information on subject areas within each institution, not just on the institution as a whole. But this stipulation is causing the most concern.

The UK steering group was alarmed by a report to the steering group on the Australian model, the Course Evaluation Questionnaire, from consultants Segal Quine Wicksteed.

It reported that in Australia, some subject areas received just one or two responses. While, for example, life sciences at Melbourne were judged by 86 students, the quality of economics at Murdoch University received just one response, compared with 15 at Sydney and 12 at Monash.

The consultants' report also caused concern about the cost of carrying out the survey. It said that if institutions handled their own mail-outs and data compilation, it could cost them £155,000 a year. The survey could be conducted less frequently than annually but information might then be out of date.

Chris Weavers, vice-president (education) at the National Union of Students, who is also a member of the steering group, said the key sticking point was how broadly the survey defined subject areas. "If you boil it down to the level of the course - which is more useful for students - you're likely to get a very low response, as some courses have only a handful of students. If you use broader subject areas, it is possible to get reasonably valid data. But the right balance has to be struck."