Determining the pass mark

?
 Determining the pass mark
Standard setting
Professional examinations are “high stakes”
Criterion referenced not norm referenced
Multiple choice, OSCEs must have standard
setting
 Standard setting
Adjust marking to a set scale where pass mark is 50%:
• Essay
Relies on judgements of every
• Viva examiner, may not be anonymous
• Long case/short case
Decide on pass mark on basis of exam content
• MCQ
Judgement on standard made
• Short answer questions by consensus, anonymous
• OSCE
Result can be scaled to fit requirements of 50% pass mark
 Setting the pass mark
University pass
Number of
mark 50%:
students 5% fail: arbitrary
norm
referenced

0 50 100
Criterion referenced: score
Objective, defensible
 Pass or fail?
determining the threshold

pass mark
 MCQ: Ebel’s Method -1
• Difficulty-Relevance decisions
– Judges read each item and assign it to one of the
categories in the classification table
• easy – medium – hard
• essential – important – acceptable
(must know – useful to know – nice to know)
– They make judgments about the percentages of
items in each category that borderline test-takers
would have answered correctly
– Calculate passing score
 Ebel’s method
Easy Medium Hard

Essential 90% 70% 50%

Important 80% 60% 40%

Acceptable 70% 50% 30%

After: Case SM and Swanson DA (1988) Constructing Written Questions for the Basic and
Clinical Sciences; Ebel RA (1979) Essentials of Educational Measurements p 337-342
 Standard setting: Ebel’s method
A class is asked to produce a map of Africa.
Which of the following countries has the
largest land area?:
a. Djibouti
b. Ethiopia
c. Puntland
d. Somalia
Easy – medium – hard
e. Somaliland Essential – important - acceptable
 Standard setting: Ebel’s method
A 70 year old lady presents with a history of fever and
left sided chest pain for one month. Examination of the
respiratory system shows decreased chest movements,
stony dull percussion note and absent breath sounds on
left side. What is her chest X-ray likely to show?:
a. Collapse
b. Consolidation
c. Fibrosis
d. Pleural effusion
e. Pneumothorax
Easy – medium – hard
Essential – important - acceptable
 Standard setting for the OSCE
• Angoff method
• Borderline regression
 Standard setting - Angoff
• Number of judges with teaching experience
• Look at each question/item in a question
• Each judge rates likelihood that a minimally
competent/borderline student will get it right
• Discuss judgements, each judge scores again on the
basis of discussion - may change or not
• Average score for judges = passing index for
question
• Average of all passing indices = pass mark for paper
 Standard setting: modified Angoff
• Judges decide what proportion of borderline
students would get a whole question right
or
• Decide yes or no for a borderline student at
each question rather than a percentage
• Review own decision
• Discuss and agree
• Produce overall pass mark
Compromise: Hofstee’s method
– Select the judges
– Discuss
• Purpose of the test
• Nature of the examinees
• What constitutes adequate/inadequate knowledge
• Ask the judges to answer four questions:
1. What is the minimum acceptable cut score?
2. What is the maximum acceptable cut score?
3. What is the minimum acceptable fail rate?
4. What is the maximum acceptable fail rate?
After the test is given, graph the distribution of scores and
select the cut score
 Motor system of the lower limbs examination
2 minutes left examiner to ask “tell me your findings from the examination”
global rating for ability to examine the motor system
Achieved Not achieved
1 Introduction and orientation and takes consent
2 Positions and exposes patient appropriately
3 Ensures that patient is not in pain
4 Inspects for abnormal posture, deformities. Inspects for shape,
symmetry of muscles, fasciculation
5 Tests tone adequately on both sides
6 Tests power both sides for: hip flexion, extension
7 Power: knee flexion and extension
8 Power: plantar flexion and dorsiflexion foot and big toe
9 Elicits knee jerk appropriately on both sides
10 Elicits ankle jerk appropriately on both sides
11 Tests for clonus
12 Tests for plantar responses
13 Tests for co-ordination
14 Assesses gait appropriately
15 Presentation of findings
16 Detects correct findings
Very good good Satisfactory Unsatisfactory Poor
5 4 3 2 1
Borderline regression
Achieved Not achieved
1 Introduction and orientation and takes consent
2 Positions and exposes patient appropriately
3 Ensures that patient is not in pain
4 Inspects for abnormal posture, deformities. Inspects for shape,
symmetry of muscles, fasciculation
5 Tests tone adequately on both sides
6 Tests power both sides for: hip flexion, extension
7 Power: knee flexion and extension
8 Power: plantar flexion and dorsiflexion foot and big toe
9 Elicits knee jerk appropriately on both sides
10 Elicits ankle jerk appropriately on both sides
11 Tests for clonus
12 Tests for plantar responses
13 Tests for co-ordination
14 Assesses gait appropriately
15 Presentation of findings
16 Detects correct findings

outstanding very good pass borderline fail

 Standard setting
• Pass/fail decisions must be defendable
• Clear pass/fail criteria set up in advance
• Any mark adjustments must be determined in
advance, in regulations
• Individual examiners must be able to trust the
system
 Combining assessments
clinical + written + in course
• Different approaches
– Compensatory
– Partially compensatory
– Conjunctive model
• Weighting of clinical v written etc.
 Combining assessments
3 tests – who should pass this examination?
pass 1 2 3 4 5

Knowledge test MCQ 50 60 90 45 50 50

Clin Skills test OSCE 50 45 60 50 45 50

Attitudes ICA 50 45 20 50 60 50

total 150 150 170 145 155 150

Result? Result? Result? Result? Result ?

 Combining assessments:
compensatory
3 tests
pass 1 2 3 4 5

Knowledge test MCQ 50 60 90 45 50 50

Clin Skills test OSCE 50 45 60 50 45 50

Attitudes ICA 50 45 20 50 60 50

total 150 150 170 145 155 150

compensatory
 Combining assessments:
compensatory
3 tests
pass 1 2 3 4 5

Knowledge test MCQ 50 60 90 45 50 50

Clin Skills test OSCE 50 45 60 50 45 50

Attitudes ICA 50 45 20 50 60 50

total 150 150 170 145 155 150

compensatory pass pass fail pass pass

 Combining assessments:
partial compensatory
3 tests
pass 1 2 3 4 5

Knowledge MCQ 50 (min 45) 60 90 45 50 50

test
Clin Skills test OSCE 50 (min 45) 45 60 50 45 50

Attitudes ICA 50 (min 45) 45 20 50 60 50

total 150 150 170 145 155 150

Partial
compensatory
 Combining assessments:
partial compensatory
3 tests
pass 1 2 3 4 5

Knowledge MCQ 50 (min 45) 60 90 45 50 50

test
Clin Skills test OSCE 50 (min 45) 45 60 50 45 50

Attitudes ICA 50 (min 45) 45 20 50 60 50

total 150 150 170 145 155 150

Partial pass fail fail pass pass

compensatory
 Combining assessments:
conjunctive
3 tests
pass 1 2 3 4 5

Knowledge test MCQ 50 60 90 45 50 50

Clin Skills test OSCE 50 45 60 50 45 50

Attitudes ICA 50 45 20 50 60 50

total 150 150 170 145 155 150

conjunctive
 Combining assessments:
conjunctive
3 tests
pass 1 2 3 4 5

Knowledge test MCQ 50 60 90 45 50 50

Clin Skills test OSCE 50 45 60 50 45 50

Attitudes ICA 50 45 20 50 60 50

total 150 150 170 145 155 150

conjunctive fail fail fail fail pass

 Combining assessments
3 tests – who should pass this examination?
pass 1 2 3 4 5
Knowledge test MCQ 50 60 90 45 50 50
Clin Skills test OSCE 50 45 60 50 45 50
Attitudes ICA 50 45 20 50 60 50
total 150 150 170 145 155 150
Result? Result? Result? Result? Result ?
compensatory pass pass fail pass pass
partial pass fail fail fail pass
compensatory
conjunctive fail fail fail fail pass
 pass 1 2 3 4 5

Knowledge test MCQ 50 50 50

Clin Skills test OSCE 50 50 49.9

Attitudes ICA 50 50 50

total 150 150 149.9

Result?
 pass 1 2 3 4 5

Knowledge test MCQ 50 50 50 50

Clin Skills test OSCE 50 50 49.9 49.8

Attitudes ICA 50 50 50 50

total 150 150 149.9 149.8

Result? Result?
 Adjusting the pass to 60%
maximum150 150

MC pass =
105 = 70%
60%
= 90

0 0
 Scaling the mark expressed as percentage
100 100
Range Range
available = available =
MC pass = 30 40
70%
60%

0 0
 Scaling the mark expressed as percentage
100 100

MC
pass
60

0 0
 Scaling the mark as percentage
100 100
Range Range
available = available =
MC pass = 70% 30 40

60%

0 0
 100 100
Range available Range available
= 30 = 40
MC pass = a
b
70%
60%

a/30 = b/40
b = a x 40/30
Adjusted mark = 60 + b
 150 150
Range Range
available = available =
MC pass = 45 60
105 = 70%
60%
= 90

0 0
 150 150
Range Range
a available = available =
MC pass = 45 60
105 = 70%
60%
= 90

•Value a - 105 (=
amount above
pass value)
•x 60/45 to scale
it to the required
pass range
•changed to %
•added to 60 0
0
 Scaling the mark to 60% for a fail score
150 150

convert mark to %
100 x a/max = a%
MC pass =
b = a% x regulation/actual
105 = 70%

60%
100 x 70/150 = 46.7%
= 90
46.7% x 90/105 = 40%
b or
a = 70 46.7% x 60/70 = 40%

0 0
 Now it’s your turn!
total mark for OSCE = 150
criterion referenced pass mark = 120/150 = 80%
University pass mark = 60% = 90/150

• candidate A = 120/150 (80%) mark?

• candidate B = 150/150 (100%) mark?
• candidate C = 135/150 (90%) mark?
• candidate D = 60/150 (40%) mark?
 Giving feedback on assessments
Learning Outcomes

• To understand why we give feedback and what it

should achieve
• To recognise how to give effective feedback
• To gain an understanding on the effect of giving
feedback to the assessment process
• To have an awareness of the different ways to provide
feedback and rationale for using each method
• To be able to ascertain how the feedback has been
received