Azam Afzal Assessment Final Paper on a specific measurement method used in medical education.

This paper should be approximately 2000-2500 words and should show evidence that the participant has integrated practice (i.e., practical approaches and how to implication) with research and theory. The document must represent an individual effort.

Objective Structured Teaching Exercise (OSTE) in the MCPS-HPE

Introduction: Lesky and Wilkerson first used standardized students for assessing and developing perceptorship which they coined the Objective Structure Teaching Exercise (OSTE).1 In 1996, Prislin and colleagues in the university of Irvine California studied the characteristics of the OSTE and found it having acceptable psychometric standards of assessing teaching skills and faculty development.2 Rationale: The effectiveness of evaluations to assessing teaching skills has been found to be dependent on a teachers charisma or communication style than actual teaching skills.3 Dunnington and DaRosa stated in their work, the need for developing a course to improve health professional teaching skills. They discussed, that faculty development courses should incorporate active learning with opportunities for practicing skills and the provision of feedback as integral qualities to the course. Furthermore in their research they assessed resident teachers teaching skills with the OSTE as they found it to be valid and reliable.4 Based on the previously mentioned quality metrics of the OSTE and its congruency with adult learning principles the OSTE would be an obvious choice for a faculty development program such as the MCPS-HPE. OSTE in the MCPS-HPE: Evidence based practice in medical education is of great importance, hence following description of the OSTE implementation in the MCPS-HPE, has been provided along with evidence to support it. OSTE Development: The OSTE stations were developed as scripted teaching scenarios which included review of instructional material, rapport building, providing effective feedback, post-examination analysis, application of metacognitive strategies and quality assurance. The process for OSTE development entailed, identifying teaching competencies, developing realistic teaching scenarios, developing an

assessment protocol for the competency and finally instructional guidelines for simulated student/faculty member training.5 OSTE Organization: OSTE Stations were selected according to the table of specification of the MCPS-HPE. Once stations were selected a map of the OSTE circuit was developed to aid in the organizing the examination. Before implementing each station is reviewed for spelling errors, instructions for students are clear and required material is provided. OSTE Conduction: Before the summative OSTE learners had already been given a mock OSTE to acclimatize them to the assessment method; after which a debriefing session is held to discuss the stations and review key teaching points and provide them with a formative feedback for learning. Learners interact with standardized student/faculty in a simulated scenario.6 A 15 station summative assessment was used, of these 13 are active and two were rest stations. Performance of students was marked by examiners who were also the simulators.7 Scoring is done on a peer reviewed analytical rating scale developed for each station.8 The pass fail criteria used was 60% for each station. This was used because the fixed percentage standard setting is a method best known for its simplicity and ease of use. Unfortunately the limitation of this method is that it produces relative standards independent of the test content and how much the examinee knows.9 Analysis of Assessment Data: A student satisfaction survey was carried out after the OSTE. It revealed that students found the experience challenging yet enjoyable, as the scenarios were relevant and realistic10 which helped motivate students to learn from the OSTE. They also found the time each allocated for each station was appropriate.11 To ensure content validity of the OSTE summative assessment for the MCPSHPE the assessment was developed using an examination blueprint.12 Analysis of the students results found that the Cronbachs alpha of the assessment was 7.94. Hence an acceptable measure of internal consistency was found in the assessment.13 However when looking at the alpha if item was deleted, 3 stations showed values which indicated that these stations may require review.

Analysis was done to determine the internal consistency of the assessments, i.e. inter item correlations were calculated as Cronbachs alpha alone may not an accurate measure of reliability this assessment method.14 Inter item analysis was performed using each OSTE station as an item and computing the corrected item-total correlation for each station. This helps determine how each item (station) correlates with each in the assessment. 3 OSTE stations negatively impacted reliability. The item correlation was rerun without the problem items and negative correlations were not seen. The content and nature of each problem station was also reviewed to determine the possible cause of the problem.15 The scoring of the OSTE is out of 10 for each station. The mean score varied between 5.46 and 8.63 (see table1 in appendix). Strengths: The OSTE is a reliable, valid and acceptable assessment of teaching skills health professions education, 16 which encourages participants to seek further educational training. It has been observed that increase in the number of stations and testing time improves OSCE reliability,17 as the OSTE is a variant of the OSCE this may hold true for the OSTE as well. The OSTE provides a unique opportunity for health professions educationists to practice their teaching skills and receive direct, immediate feedback for enhancement of teaching skills. This makes the OSTE beneficial for faculty development purposes.18 An important competence for health professions educationists is communication skill. Like the OSCE the, OSTE is a valid tool which may be used to assess this skill in a simulated setting while interacting with the simulator. Components for effective communication such as rapport building, active listening and even non verbal aspects of communication may be assessed in the OSTE in a short amount of time.19,20 The OSTE also helps program administrators evaluate the effectiveness of faculty development curricula as it can create an atmosphere for realistic practice and offers an alternative to evaluations allowing greater sampling and standardization of teaching cases. It also allows for greater reliability in scoring because of the use of an analytical rating scale for scoring student performance.21 A key strength in the OSTE is the use of analytical rating scales for scoring student performance. The rating scale, especially when scored by subject experts has been found to have greater inter-station reliability and concurrent validity as compared to a checklist.22 Although it has been found;
3

that checklists have the advantage of yielding similar reliability in the hands of non-trained assessors as subject experts.23 Checklists do not take into account the level of expertise of the examiner and since the MCPS is a post graduate exam that aims to develop subject specialists in health professions education; the OSTE utilizes the subject expertise of faculty members as well as other subject specialists in this field as assessors.24 OSTE stations in the MCPS-HPE may cover multiple domains of learning. Looking at the work of Regehr and associates; the use of a checklist can make the scoring too content specific which would make the scoring very restrictive and decrease effectiveness of the assessment. This was evidenced by the authors who found a decrease in generalizability scores across stations and throughout the assessment. This problem is not encountered with a rating scale. 25 Limitations: Developing and implementing can be time and resource intensive,26 especially when it comes to standardized simulators this requires training simulators to be standardized students for the OSTE.27 The analytical rating rubric although it increases the objectivity of the item, is still less objective as compared to a rating scale. Although according to some authors both are equally suited for the purpose.28 Since the OSTE is essentially an OSCE modified for teaching scenarios then it is logical that it also inherits the same limitations of the OSCE. Such as the question of validity being compromised if a complex skill is divided into multiple minor tasks; although reliability may be enhanced by such an action. According to Miller the OSTE tests a students ability to perform the task, yet under simulated conditions the domains that are assessed are often specific to a certain problem rather than to a skill that may be generalized across teaching problems.29 Also, in the OSTE the student is under scrutiny of the examiner so handles ethical tasks appropriately, this does not guarantee if the student will actually perform in the same manner in an unobserved real life situation. 30 A key concern is the confidentiality of the OSTE bank; several studies have failed to show higher scores on reuse of performance-based assessments. Stillman et al examined the effect on the mean scores of successive testing of multiple student/resident populations.31 Though these data suggest that security is not significantly compromised by administering performancebased examinations at different times, the authors discuss that the low-risk stakes of their examinations are well a the dispersal of test takers may have resulted in less dissemination of the test questions. While Cohen et al found
4

an increase in mean scores of reused OSCE station. They suggested that candidates scores may improve as a result of concentration of their efforts on subjects repeatedly used in the examination rather than of sharing the answer key.32 Recommendations: The rating scale although of great benefit may require modification in select components of the OSTE. Incorporation of a 5 point rating scale might be useful as this would increase the objectivity of the assessment; it would also help intra-class correlation determination of the stations and hence help determine the OSTE reliability with greater accuracy. The use of a the 5 point rating scale would also help assessors save time during examinations as compared to the use of the rating scale currently used in which assessors have to write down marks for each component.33 OSTE Bank development activities should be more frequent so that OSTE station re-use in assessments is minimized and OSTE bank confidentiality is maintained. A justified standard setting method could be adopted such as the Angoffs method which also produces absolute standards and focuses attention on each of the questions individually.34 Conclusion: The OSTE is a reliable, valid method of assessment for faculty development programs. The quality of this assessment method may be improved using standardized simulators, an objective rating instrument such as a 5 point rating scale, blueprinting and review of assessment metrics. 35

Appendix
Mean Static Interactive Interactive Static Interactive Interactive Interactive Interactive Interactive Static Interactive Interactive Interactive 5.667 6.800 8.000 5.467 7.467 7.033 8.200 7.733 8.467 6.600 8.000 8.633 8.333 Std. Deviation 1.4475 1.8107 .5000 1.4075 1.0083 .8121 1.2071 1.0998 .7432 .7368 1.0690 1.2459 1.1751 Alpha if Item Deleted .765 .755 .782 .785 .778 .777 .825 .763 .766 .801 .756 .797 .781 Squared Multiple Correlation R2 .988 .911 .970 .947 .909 .961 .887 .909 .973 .980 .892 .926 .668

Sample OSTE station:

OSTE COVER SHEET Specialty: Health Professionals Education

SERIAL NO: TOPIC: Constructive feedback COMPETENCE TO BE ASSESSED: Ability to give constructive feedback by sandwich technique TYPE OF STATION: INTERACTIVE/ STATIC RESOURCES REQUIRED: (Please write number of item required where necessary and put a tick mark before it chairs Any other, please write the item and the number required below:
_____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ ____________________________________ NAMES OF EXAMINERS DEVELOPING THE STATION: DATE THAT STATION WAS DEVELOPED: (TO BE FILLED IN BY THE EXAMINATION DEPARTMENT) Date when previously used Difficulty index Discrimination index

OSTE INSTRUCTIONS FOR EXAMINERS

Specialty: Health Professionals Education

SERIAL NO: TOPIC: COMPETENCE TO BE ASSESSED: TYPE OF STATION: INTERACTIVE/ STATIC INSTRUCTIONS AS GIVEN TO CANDIDATE IN HIS / HER INSTRUCTION SHEET:

A surgeon who is your immediate junior asks you to attend his class so that you may comment on his teaching skills. The surgeon is teaching final years students and the topic of his lecture is hernia.

You attend his lecture and feel that the surgeon is unable to maintain eye contact with his students, he is also unable to answer simple questions asked by the students.

QUESTIONS TO BE ASKED FROM THE CANDIDATE ARE ON THE SCORING SHEET (IF APPLICABLE)

You have called him in your office to give him feedback regarding his session.

OSTE SCORING SHEET Specialty

STATION NO: TOPIC: Constructive Feedback EXAM CENTRE: COMPONENT/ QUESTIONS

ROLL NO: DATE: KEY FEATURES/ AGREED ANSWERS RATING SCALE COMPONENT SCORE EXCELLENT GOOD ADEQUATE INADEQUATE POOR

Rapport Building

1.

Welcome and introduce self.

Listening & comments

1. 2. 3. 4. 1. 2. 3.

Behavior

Asks questions about his presentation and Nodding. Reinforce positive points. Softly give negative points. End with a positive and encouraging comment. Facial expression. 1. Eye contact with Tone. gentle, nonAttitude threatening way. 2. Non-judgmental one-to-one manner. 3. Listening actively and giving positive cues.

TOTAL

10

Prompt questions (if any)

Prompt questions (if any)

EXAMINERS NAMES _____________________________ EXAMINERS SIGNATURES________________________

_____________________________ ____________________________

PROCEDURE INSTRUCTIONS FOR CANDIDATES Specialty: Health Professionals Education

STATION NO: TOPIC: TIME ALLOWED: DESCRIPTION OF THE TASK TO THE CANDIDATES (information and instructions)

10

References:

Lesky LG, Wilkerson L. Using standardized students to teach a learnercentered approach to ambulatory precepting. Acad Med 1994;69(12):955957 Prislin MD, Fitzpatrick C, Giglio M, Lie D, Radecki S. Initial experience with a multistation objective structured teaching skills evaluation. Acad Med 1998;73(10):1116-8 Irby DM. Teaching and learning in ambulatory care settings: A thematic review of the literature. Acad Med 1995;70(10):898-931.
3

Dunnington GL, DaRosa D. A prospective randomized trial of a residents-as-teachers training program. Acad Med 1998;73(6):696-700. Zabar,S, Hanley,K, Stevens,D,Kalet, A, Schwartz, M, Pearlman, E, Brenner, Kachur, E, Lipkin, M. Measuring the Competence of Residents as Teachers. JGIM;2004; 19(5):530.
5

Gelula MH, Yudkowsky R. Microteaching and Standardized Students Support Faculty Development for Clinical Teaching. Acad Med 2002;77(9):941.
6

Newble D, Swanson DB. Psychometric characteristics of the objective structured clinical test. Med Educ 1988; 22(4): 325-334.
7

Gronlund N. E. and Waugh C. K. Performance assessments. Assessment of Student Achievement. Upper Saddle River, NJ: Pearson Education 2008.
8

Norcini JJ. Setting standards on educational tests. Med Educ 2003; 37:464-469.
9

Kaufman DM. Applying educational theory in practice. In: Cantillon P, Hutchinson L, Wood D, editors. ABC of learning and teaching medicine. London; BMJ publishing; 2003; p1.
10

Kaufman A. Dodge T. Student perceptions and motivation in the classroom: exploring relatedness and value. Soc Psychol Educ. 2009; 12(1):101-112.
11

11

Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003;37:830837.
12

13

Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004;38:10061012. Pell G, Fuller R, Homer M, Roberts T. How to measure the quality of the OSCE: A review of metrics AMEE guide no. 49. Med Teach 2010;32:802811.
14

Auewarakul C, Downing SM, Praditsuwan R, Jaturatamrong. Item Analysis to Improve Reliability for an Internal Medicine Undergraduate OSCE. Advances in Health Sciences Education 2005 10:105113
15

Morrison EH, Boker JR, Hollingshead J, Prislin MD, Hichcock MA, Litzelman DK. Reliability and validity of an Objecive Structured Teaching Examination for Generalist Resident Teachers. Acad Med 2002;77(10):S29S32
16

Newble D, Swanson DB. Psychometric characteristics of the objective structured clinical test. Med Educ 1988; 22(4): 325-334.
17

Stone S, Mazor K, Devaney-O'Neil S, Starr S, Ferguson W, Wellman S, Jacobson E, Hatem DS, Quirk M. Development and implementation of an objective structured teaching exercise (OSTE) to evaluate improvement in feedback skills following a faculty development workshop. Teach Learn Med 2003;15(1):7-13
18

Kalet et al. Teaching Communication in Clinical Clerkships: Models from the Macy Initiative in Health Communications. Acad Med 2007; 79(6):511520.
19

Ishikawa H, Hashimoto H, Kinoshita M, Fujimori S, Shimizu T, Yano E. Evaluating medical students' non-verbal communication during the objective structured clinical examination. Med Educ. 2006 Dec;40(12):1180-7.
20

Gronlund N. E. and Waugh C. K. Performance assessments. Assessment of Student Achievement. Upper Saddle River, NJ: Pearson Education 2008.
21

12

Regehr G, MacRae H, Reznik RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance in on an OSCE format examination. Acad Med 1998; 73(9): 993-997.
22

Moineau G, Power B, Pion AJ, Wood TJ, & Humphrey-Murto S. Comparison of student examiner to faculty examiner scoring and feedback in an OSCE. Medical Education 2011; 45: 183191
23

Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ 2003;37:10121016.
24

Regehr G, Freeman R, Hodges B, Russell L. Assessing the generalizability of OSCE measures across content domains. Acad Med 1999; 74(12):1320-1322.
25

Post RE, Quattlebaumn RG, Benich JJ. Residents-as-Teachers Curricula: A Critical Review. Acad Med 2009; 84(3): 374-380.
26

Zabar,S, Hanley,K, Stevens,D,Kalet, A, Schwartz, M, Pearlman, E, Brenner, Kachur, E, Lipkin, M. Measuring the Competence of Residents as Teachers. JGIM;2004; 19(5):530.
27

Norman D. Post graduate assessment reliability and validity. Trans. J. Coll. Med. S. Afri. 2003; 47: 71-75.
28

Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; 65: 563-67.
29

Wass V, Vleuten CV, Shatzer J, Jones R. Assessment of clinical competence. Medical Education Quartet. Lanect 2001; 357: 945-49.
30

Stillman PL, Haley HA, Sutnick AI, Philbin MM, Smith SR, O Donnell J, and Pohl H. Is test security an issue in a multi-station clinical assessment? A preliminary study. Academic Medicine 199; 66S25-S27.
31

Cohen R, Rothman A, Ross J, and Poldre P. Security Issues in Standardized- Patient Examinations. Academic Medicine 1993; 68:S73-S75.
32

Throndike RM. Assessing Process. In: Measurement and evaluation in Psychology and education. 6th ed. Upper Saddle River, New Jersy: PrinticeHall; 1997.
33

13

Norcini JJ. Setting standards on educational tests. Med Educ 2003; 37:464-469.
34

Pell G, Fuller R, Homer M, Roberts T. How to measure the quality of the OSCE: A review of metrics AMEE guide no. 49. Med Teach 2010;32:802811.
35

14