A: The exam you devised - if you've ensured that students haven't been able to predict the questions, you've given them time to complete the paper and you haven't tried to catch them out, says Harriet Swain
You are writing an exam question. Do you a) use the same one as last year, b) use it as a chance to experiment with a pet pedagogical theory, c) dash off a few paragraphs and stick a question mark at the end?
None of the above, says Jude Carroll, deputy director of the Centre for Excellence in Teaching and Learning at Oxford Brookes University, which focuses on assessment.
She says that if you want to avoid your question being anticipated, you need to avoid repeating something that has been asked before. You also need to take time to think carefully about the kinds of questions that will be impossible to predict. "Students are getting better and better at question-spotting and preparing and memorising answers," she warns.
Linda McDowell, director of Oxford Brookes' sister centre at Northumbria University, says you also need to avoid matching questions too closely to the topic of a particular lecture, which would encourage students simply to regurgitate what they have learnt.
But while you should try to be original, it is not fair to take this too far. Carroll says that if you want to set a different kind of question, you need to make sure that students have had a chance to familiarise themselves with the format.
Phil Race, co-author of 500 Tips on Assessment , says: "Don't experiment with students' futures. An exam question should be based on something you've already tried out in the safety of a formative assessment context, so that you have a good idea how it is going to work and are sure that there won't be unexpected consequences when used in an exam."
Carroll says you also have to make sure your question is worth answering, and not a puzzle in itself. Many questions rely on implied understanding.
For example, a question that makes a statement and then simply states "discuss" assumes that the student will not turn to his or her neighbour and start making conversation. But you need to be sure that all students will interpret it that way. "As the group of students becomes more diverse, we need to be more clear about what we want them to do," she says.
Race advises saying to yourself, "What I really mean is...", before jotting down the first draft of each sentence. Short sentences and bullet points or separate parts are easier for tense students to handle. He says you should not try to be too clever or to catch students out. If your language contains hidden subtleties, your question could be deemed unlawful under the Special Educational Needs and Disability Act by discriminating against certain categories of student - those with English as a second language, for example.
Andrew Watts, director of the Cambridge Assessment Network, which provides professional development for people working in assessment, says you should try to use active rather than passive tenses and make sure that if you are presenting a sequence of ideas that they follow in a logical order.
McDowell says you need to avoid using examples of places, people or events that some students may not understand. While you might assume that everyone knows about certain contemporary events, she warns that it is always possible that some students might not understand the reference.
Asking colleagues what they think the question means is a good way to find out whether it has ambiguities you have not spotted. Asking them to answer the question is even better, Race says. It will alert you in advance to the ways different students might approach it, and gives you the opportunity either to accommodate these different approaches in your marking scheme or to make your intended approach clearer.
Watts says you have to be clear that the question is testing exactly what you want it to. A maths question that involves large amounts of text would be examining reading as well as mathematical ability, for example.
He says that thinking about the purpose of the question is essential before you start. You have to be clear first about whether you are testing the students' progress to diagnose problems, or testing achievement. Trying to make a question do too much is usually a mistake, he warns. McDowell advises thinking about whether you want to test coverage of a range of topics or whether you are aiming for students to demonstrate more academic skills, such as analysis and synthesis, which would demand a lengthier answer and more time for the question. She suggests starting with a shopping list of topics and skills you want to examine before thinking about the question itself.
Race says you should keep your intended learning outcomes in mind and make sure you cover all of them rather than testing the same things over and over again.
You also need to think about the practicalities of your question, making sure students will have time to answer it. "Assessors setting problem-type questions for students often forget that familiarity with the type of problem profoundly influences the time it takes to solve it," warns Race.
Meanwhile, Carroll says you have to bear in mind the time constraints on markers too. She says that time invested in setting the question often pays off in the end by significantly reducing the time spent marking them.
Watts says that one of the most intellectually demanding aspects of preparing a question is writing the marking scheme. You must try to ensure that the same question tackled by the same student would give the same result no matter when it was attempted or who was marking it.
Once you have done all this, Race says you need to proofread your exam questions carefully. "Be aware of the danger of seeing what you meant, rather than what you actually wrote," he warns. Leave that kind of problem to those sitting your exam.
Further information
500 Tips on Assessment , second edition, by Phil Race, Sally Brown and Brenda Smith, Routledge, 2004.
The Cambridge Assessment Network: www.cambridgeassessment.org.uk/network/
TOP TIPS
Use active rather than passive tense
Consult your colleagues
Be clear and do not assume too much implied knowledge
Check your work