HOW LIKELY is it that you will fail an examination even if you are competent? Such a question can be uncomfortable but it is a question worth asking and should, perhaps, be asked more often. In fact whenever a system is set up on which people's lives depend the question should be asked. Unfortunately it seems only to come to the fore in times of crisis. (When the innocent are released from jail or when smear tests falsely suggest cancer, for example.)
The recent Ofsted inspections of initial teacher training institutions (THES, December 12) are played for particularly high stakes and the system that was set up merits a closer look. The new inspection system required that each institution be rated in 14 different areas. If it failed just one of them it failed overall. It does not require much thought to realise that this is potentially problematic. Although throwing a six is quite unlikely, if you keep throwing the die you will get a six sooner or later.
I estimate that a very high-quality initial teacher training institution would have just a 0.2 per cent chance of failing in any single area if the inspection team were a little unreliable. This translates into a 3 per cent chance of failing overall. Perhaps such a risk is acceptable, perhaps not.
More worrying is the inspection of an unexceptional institution that is just satisfactory in all 14 areas. I estimate that it has a 73 per cent chance of failing in at least one of the 14 areas when inspected by a very reliable Ofsted team. One might argue that this high chance of failure for a "just satisfactory" institution is tolerable, but there is more to come.
Two of the 14 areas involve Ofsted assessing students who have already been assessed as passing by the institution. If Ofsted believes any of these students to be failing then the institution fails. The system requires that the institution nominate students for this part of the inspection. They must pick out four top, four good and eight "adequate" students. These eight are to be identified as either high or low in the adequate category. In other words every institution must put forward four of their most difficult cases. The ones over which there was heart searching. Now, if a single one of these is judged to be failing during the all-too-brief visit by Ofsted the institution fails.
Not unsurprisingly this produces a fairly high chance of failure. This chance is more or less independent of the reliability of the inspection process and of the quality of the institution. When very borderline cases are inspected the judgement can go either way. I estimate that a very good institution inspected by a very reliable team would have about a 50:50 chance of failure under this regime.
This is a worrying situation. It is against natural justice but cannot be challenged in law. Surely the universities cannot tolerate this.
Peter Tymms Reader in education University of Durham (A fuller account of the estimations referred to may be found in Education Line www.leeds.ac.uk/ educol/index.html)