doi:10.1111/j.1440-1754.2011.02115.

ANNOTATION

How to write good multiple-choice questions

Dianne E Campbell1,2
1

jpc_2115

322..325

Discipline of Paediatrics and Child Health, University of Sydney and 2Department of Immunology and Allergy, The Childrens Hospital at Westmead, Sydney, New South Wales, Australia

Abstract: Medical practitioners are frequently called upon to write multiple-choice questions for high stakes examinations, such as nal year medical student assessments and specialist college accreditation examinations. The fairness and validity of these examinations depends principally on the generation of high-quality questions, which adequately sample the curriculum. The most common format for multiple-choice examinations are single best answer and extended matching type questions. This paper provides a simple guideline that will assist with the production of high-quality multiple-choice papers in these formats. Key words: examination; extended matching; medical education; multiple-choice.

Introduction
Medical practitioners are increasingly being asked to contribute multiple-choice type exam questions for the assessment of medical students and junior doctors for universities and accrediting colleges. Although it may appear supercially easy to write a few multiple-choice questions (MCQs) in ones area of specialty, many such hastily written questions perform poorly in examinations and fail to discriminate between high- and lowperforming candidates. Just like any other task in medicine, there are principles, which when followed, will vastly improve the quality of the individual question and the examination as a whole. Any examination should endeavour to be fair, valid and reliable. Reliability substantially depends on the number of items in an examination, whereas the fairness and validity of an examination rely on the quality of each individual item. This how to guide seeks to summarise the most popular MCQ formats and outlines a simple process of exam question writing that will result in the ability to generate high-quality

questions without signicantly increasing the time expenditure dedicated to these tasks.

Question Formats
Single Best Answer (SBA)
The most commonly used form of MCQs used in science, medicine and postgraduate medical assessment is SBA. These questions are often constructed in scenario form, with an introductory stem, which sets the clinical scenario, a lead-in question such as; Which one of the following is the most likely diagnosis?, and a list of ve possible responses consisting of four distractors and one correct answer. These are sometimes referred to as A type MCQs. They may test only fact recall in their most basic form; however, when written as a clinical scenario, they often test the ability to synthesise information and to some extent clinical reasoning.1 For example:

Stem
An 8-month-old boy is brought to the emergency department by his mother. She is worried as he has had a fever and cough for the past 16 h and is now struggling to breath. On examination he is alert, not centrally cyanosed with marked sub- and inter-costal recession. He has widespread wheeze and inspiratory crepitations over both lung elds.

Key Points 1 Single Best Answer and Extended Matching are the most commonly used style of multiple choice questions. 2 Following a few simple guidelines when developing questions can signicantly improve the fairness and validity of the examination. 3 Students are test wise and use well known strategies to guess correct answers in poorly written multiple choice questions.
Correspondence: Associate Professor Dianne E Campbell, The Childrens Hospital at Westmead Clinical School, Childrens Hospital Westmead, Sydney, NSW, Australia. Fax: 0298453389; email: campbell@med. usyd.edu.au Accepted for publication 13 February 2011.

Lead-in
Which one of the following is the most likely diagnosis?

Responses (one correct and four distractors)

A B C D E Laryngotracheobronchitis Epiglottitis Asthma Bronchiolitis Atypical pneumonia

322

Journal of Paediatrics and Child Health 47 (2011) 322325 2011 The Author Journal of Paediatrics and Child Health 2011 Paediatrics and Child Health Division (Royal Australasian College of Physicians)

Annotation

Annotation

Multiple true/false questions

Older style J type (sometimes called X type) MCQs are also known as multiple true/false questions. They usually have a short introductory stem such as In the management of Diabetic ketoacidosis, which of the following is correct? They may have one or more correct answers. Although these questions are the easiest to write, they are actually very difcult to write well and their use in high-stakes examinations is generally discouraged as they have been shown to not be able to discriminate well between high- and low-ability candidates (reviewed in2). They are generally either too ambiguous or, when written to remove ambiguity, too easy without good discrimination.

as part of medical student assessment. They seek to test clinical reasoning rather than just fact recall or synthesis of clinical information.5 They are written in short scenario form with a series of pieces of additional information with which the candidate must determine whether these pieces of information make a statement or diagnosis more or less likely. They have a complicated grading system which relies on a score derived from a panel of experts, with no absolute incorrect or correct answers. They are more difcult to write than conventional MCQ questions and a detailed account is beyond the scope of this guide.

Examination Performance
Many examinations, particularly those of high stakes, such as licensing, credentialing or nal-year student examinations, use statistical methods to analyse the performance of the questions themselves as well as that of the candidates. These analyses generally compare how an individual item in an assessment performs compared with the whole exam. The most commonly used method of analysis for SBA and EMQ is the Item Response Theory based on the Rasch methodology.6,7 Here, each item can be scored for its degree of relative difculty and whether it discriminates high-performing from lower-performing students as determined by a candidates performance across the entire exam, compared with a single item. For example, if a difcult question appears harder for candidates who performed well across the rest of the exam than for those who performed badly, it could be considered a poor question and may be eliminated from the exam.

Extended matching questions (EMQ)

EMQs are increasingly used in medicine and science examinations. They are relatively easy to write, have evidence for good discrimination between student abilities3 and have been shown to be superior to standard SBA in assessment of clinical reasoning.4 They are typically structured in a theme with a list of possible options (usually 810), a lead-in question/statement and at least two to three short scenarios or stems, which are matched to the correct option by the candidate. For example:

Options
A B C D E F G H I J Measles Human metapneumovirus Parvovirus Adenovirus Varicella Molluscum contagiosum Rubella Coxsackievirus B19 HHV-6 Herpes simplex virus

The test wise candidate

Many students can be considered test wise, in that they understand how to select the correct options on the basis of the structure or wording of the question rather than on knowledge basis. Student web based sites such as bored of studies (http:// www.boredofstudies.org/) have lists of tips on how to pass MCQ questions and are frighteningly accurate in what they perceive as best ways to eliminate poorly written distractors (incorrect options). Many candidates are aware of all the elements that help to make a poor or easily guessed question (as below). Inclusion of these elements will decrease the capacity of an examination to determine actual knowledge or ability of the candidate to synthesise information and reduces the validity of the examination.

Lead-in
For each of the following children with infection, select the most likely causative agent

Scenarios/stems
1 A 2-year-old girl presents with a cough, high fever and blanching erythematous, conuent, maculopapular rash on her abdomen and chest. 2 A 12-month-old infant presents with a high fever, irritability, drooling and multiple ulcers covering his oral mucosa, including hard and soft palate. 3 A 7-year-old girl presents with an itchy widespread vesicular rash which has crusted in some areas.

Compiling an examination
Assessments should be aligned with the curriculum. What is being assessed in the examination should be material that has been delivered during the course and should reect the learning objectives and activities of the curriculum. The difculty level of the question should be appropriate for level of the candidates. When compiling a whole examination, the difculty level of questions should be spread across the range of easy, average and difcult questions.
323

Script concordance
These questions are a newer form of MCQ type questions that are used in some universities in Australia, Canada and the UK

Journal of Paediatrics and Child Health 47 (2011) 322325 2011 The Author Journal of Paediatrics and Child Health 2011 Paediatrics and Child Health Division (Royal Australasian College of Physicians)

Annotation

Annotation

Approach to Writing a MCQ

SBA
Choose a theme or disease or disorder Decide whether the lead-in will be focused on basic science, history, diagnosis, investigations or management (the same scenario can be used to write a few different questions using different lead-ins and different responses, which can be used in different examinations) Write your scenario Write the correct response and then the four distractors, which should be signicantly less correct than the correct response

EMQ
Choose a theme Write your list of homogeneous options (i.e. all investigations, or diseases or medications, etc.) Write your focused lead-in Write the scenarios that will match with two to four of your options To create high-quality SBAs and EMQs that will be reliable and fair, use the guidelines below: 1 Is the topic important? The question should be an important clinical one and suitable for the level of the candidates. 2 It is too late to teach Do not write questions that attempt to teach within a question. Many experts have particular bugbears about an area, especially when they perceive that there is a common misconception or misunderstanding in the area. It is tempting to try and write questions that address this; however, invariably, these end up as poorly performing questions. Caution should be taken when attempting to write about an area in which very strong feelings are held by the writer. 3 Does it pass the cover test? A good SBA question or EMQ scenario should be structured so that the correct response can be given by an expert or highperforming candidate even if the options are covered up. When writing a question, always check that it passes this cover test. Performing the cover test often identies the presence of insufcient relevant information or ambiguities in the stem. 4 Keep the options homogenous Make sure the options all t into a similar category, such as treatment, management, diagnosis or investigation. Keep the options of similar length and exactness. Often, the nave exam writer uses more qualifying language in the correct option than in the discriminators to make it as correct as possible. This cues the students into this option as the most likely response. Make sure the correct option is not the longest. 5 Be careful of convergence of options and logical cues Make sure that the process of elimination and logic alone cannot answer the question or reduce the odds to a 1:2 or 1:3. This occurs primarily when a subset of the options is collectively exhaustive.
324

If three of ve options include the same element, one of these options is likely to be correct. If two of these three contain a further similar element, then the test-wise student is likely to guess between one of these options. 6 The lead-in should be a focused, specic question, such as: Which of the following is the next best step in management or Which of the following is the most likely diagnosis? Do not use unfocused lead-ins, such as: Which of the following concerning X is correct? 7 Do not use lead-ins that can be easily misinterpreted, negative lead-ins or double negatives, such as: All of the following are contraindicated in this condition, except: 8 Words not to use: never use never! Never and always are never correct options, whereas possibly, often or frequently are likely to be correct. Try not to use any of them! Do not reuse a word from the stem in the options, as it is likely to be the correct answer and will cue the test-wise student. Never include all of the above or none of the above as an option as this effectively converts a functional SBA or EMQ into a multiple true false in most circumstances. 9 Do not use trigger or key words For questions that require a synthesis of information, it is best to avoid trigger words which the candidates will associate by recall alone with the disorder, rather than making the diagnosis by a process of clinical reasoning. Barking for croup, paroxysm for pertussis or conuent for measles would be examples of key words, which may cue a student to the correct response without reecting any deep knowledge of the area. This is demonstrated in the example EMQ where question 1 is clearly measles and could be identied by a poor student who had recalled only that conuent is a key word association with measles. A candidate may not even understand what conuent actually describes, and as such it is preferable to describe the symptom or sign in relative lay language, especially at medical student level examination. 10 Do not be deliberately tricky or misleading Keep the scenarios concise without unnecessary or misleading or tricky information. There should be sufcient information in the scenario or stem to answer the question without seeing the options (cover test). It is not necessary to include all the information that would be obtained in a clinical situation to have a high-quality and fair question. Irrelevant or misleading information should be eliminated.

Want More?
For detailed examples and further instructions on how to write good MCQs, there are several excellent resources including the US-based National Board of Medical Examiners Constructing Written Test Questions for the Basic and Clinical Sciences.1

References
1 Case SMSD. Constructing Written Test Questions for the Basic and Clinical Sciences, 3rd (revised) edn. Philadelphia, PA: National Board of Medical Examiners, 2002. 2 McCoubrie P. Improving the fairness of multiple-choice questions: a literature review. Med. Teach. 2004; 26: 70912.

Journal of Paediatrics and Child Health 47 (2011) 322325 2011 The Author Journal of Paediatrics and Child Health 2011 Paediatrics and Child Health Division (Royal Australasian College of Physicians)

Annotation

Annotation

3 Swanson DB, Holtzman KZ, Allbee K. Measurement characteristics of content-parallel single-best-answer and extended-matching questions in relation to number and source of options. Acad. Med. 2008; 83: S214. 4 Beullens J, Struyf E, Van Damme B. Do extended matching multiple-choice questions measure clinical reasoning? Med. Educ. 2005; 39: 41017. 5 Fournier JP, Demeester A, Charlin B. Script concordance tests: guidelines for construction. BMC Med. Inform. Decis. Mak. 2008; 8: 18.

6 Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1960. 7 Bhakta B, Tennant A, Horton M, Lawton G, Andrich D. Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education. BMC Med. Educ. 2005; 5: 9.

Journal of Paediatrics and Child Health 47 (2011) 322325 2011 The Author Journal of Paediatrics and Child Health 2011 Paediatrics and Child Health Division (Royal Australasian College of Physicians)

325