Worries over reliability delay results of satisfaction survey

Results from the first national student satisfaction survey are unlikely to be available until summer 2005, almost two years later than promised by the government, amid continuing concerns about the validity of its judgements.

The government's higher education white paper states that the views of students will be published in a national annual survey "available for the first time in autumn 2003". But the survey's steering group is concerned that the findings may not be reliable at subject level unless a large number of responses is received.

Its interim report, released this week, reveals that the pilot exercise conducted earlier this year achieved a response rate of about 40 per cent.

The group says in the report: "We do not believe that a response rate of this order would enable the survey to produce reliable results.

"High response rates will be vital in reducing the margins for error, improving reliability and allowing results to be reported at a suitably detailed level."

The group is so concerned about getting a sufficient response rate that it calls for students to be surveyed before they complete their final year of study, for fear that they may not bother filling in a questionnaire once their course is over. Doing so "better meets the public information need and better achieves the purpose of the exercise", the report says.

It dismisses concern about gathering feedback halfway through students'

final year when courses are not completed: "Evidence suggests that it is not necessary, in terms of the validity of feedback, to wait for completion."

There is still uncertainty about the way in which results will be published.

The report acknowledges the danger of prospective students and advisers seizing on apparent differences between institutions that are not statistically significant. "We are concerned that this exercise must produce results that are resistant to such misinterpretation," it states, but adds that they must be clear and accessible at the same time.

More than 60 institutions volunteered for the pilot, and 22 were selected to participate. They included Birmingham, Luton and Nottingham universities, Canterbury Christ Church University College, the Southampton Institute and the Royal Northern College of Music.

A Higher Education Funding Council for England spokesman said the results of the 15,674 responses were still being analysed and would be released next month. The report states that the average response across the pilot was "positive".

About 70 per cent of students used the open-ended question, which the report says appears to have provided a rich source of data that could be used for a number of purposes.

The students' views fell into 29 categories, ranging from overall level of support, variation in teaching standards and staff availability, to workload, stress and social life.

About £400,000 was spent on the pilot - up to £13,500 per institution. Costed models for the full-scale survey are not yet available, but the group warns that decisions on the methodology to be used "should not be driven by cost considerations".

Two paper-based versions of the survey, one randomly ordered and one structured, along with a structured web version, were produced for the pilot.

The report notes that email can prove to be a very low-cost alternative collection method, particularly if the survey is conducted before completion.

Most institutions opted for the structured paper survey and for Hefce to mail them to students. A variety of follow-up methods was used, including resending the survey, postcards and telephone calls.

The report can be read at http:///iet.open.ac.uk/nss/download/interim_report.pdf