Research papers

Assessment of student performance in problem-based learning tutorial sessions

lez, Jose Antonio Rojas-Ramirez, Rosamar{a Valle, lleana Petra, Adrian Mart{nez-Gonza Sara Morales-Lopez & Beatriz Pin a-Garza ~

1,2 Objectives To assess student performance during tutorial sessions in problem-based learning (PBL). Design A 24-item rating scale was developed to assess student performance during tutorial sessions in problem-based learning (PBL) as conducted during the pre-clinical years of Medical School at the National Autonomous University of Mexico. Items were divided into three categories: Independent study, Group interaction and Reasoning skills. Fourteen tutors assessed 152 rst and second-year students in 16 tutorial groups. An exploratory factor analysis with an Oblimin rotation was carried out to identify the underlying dimensions of the questionnaire. Setting Medical School at the National Autonomous University of Mexico. Subjects Medical students.

Results Factor analysis yielded four factors (Independent study, Group interaction, Reasoning skills, and Active participation) which together accounted for 766% of the variance. Their Cronbach reliability coefcients were 095, 083, 094 and 093, respectively, and 096 for the scale as a whole. Conclusions It was concluded that the questionnaire provides a reliable identication of the fundamental components of the PBL method as observable in tutorial groups and could be a useful assessment instrument for tutors wishing to monitor students' progress in each of these components. Keywords Problem-based learning; medical students; undergraduate medical education, *standards; educational measurement, *standards. Medical Education 1999;33:818822

Introduction
The Faculty of Medicine at the National Autonomous University of Mexico (UNAM), founded in 1579, is the oldest medical school in America. It has a total undergraduate population of 5570 students and annually admits approximately 950 students. The undergraduate programme has a duration of six years (the rst two include mainly basic sciences followed by two years of clinical medicine, a year of internship and a year of medical practice in a community. A traditional, lecture-based approach is generally used in most of the courses. However, the latest curriculum revision, carried out in 1993, suggests the use of innovative teaching methods in medicine, such as problem solving, tutorial teaching, and interdisciplinary learning.1
n Facultad de Medicina, Direccion General de Evaluacio Educativa, n Coordinacion del Sistema de Universidad Abierta y Educacio a noma de Mexico. Distancia, Universidad Nacional Auto n Correspondence: Dr Rosamara Valle. Direccion General de Evaluacio Educativa. Edicio de los Consejos Academicos, Circuito Exterior, Ciudad Universitaria, Mexico, DF, 04510

Educational strategies based on problem-based learning methods have been used in some of the courses in the rst four years with the following objectives: (a) introducing a self-learning method using a problemsolving approach (b) promoting active student participation and (c) helping students to acquire reasoning and group interaction skills. To achieve these goals, students assemble in tutorial groups of nine assisted by one staff tutor and are given a medical problem to analyse. The rst session begins with a case presentation followed by clarication of concepts and facts that are new to the students. Students are then asked to identify and analyse the problem, and attempt to formulate hypotheses based on their previous knowledge. After this, students determine learning objectives and resources required to obtain the necessary information. After a period of 3872 h during which they locate and collect information, students meet with their tutor for a second session to present their ndings, analyse the information they have gathered and discuss the case. This method is based on problem-based learning strategies as proposed by Schmidt,2 involving the

818

Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:818822

Assessment of student performance in PBL tutorial sessions

R Valle et al.

819

following seven steps: (l) clarifying terms and concepts not readily comprehensible; (2) dening the problem; (3) analysing the problem; (4) drawing up a systematic inventory of the explanations deduced from step three; (5) formulating learning objectives; (6) collecting additional information outside the group and (7) synthesizing and testing the newly acquired information. The UNAM's Faculty of Medicine has felt it necessary to assess students' progress during tutorial sessions and the achievement of PBL objectives. According to Hay,3 the evaluation of PBL students should be based on actual performance in their learning setting. This is where they should learn to make relevant hypotheses, identify their learning needs, use appropriate learning resources, provide evidence of learning and act as a responsible team member of the tutorial group. However, it has proved difcult for tutors to judge whether their students have acquired the desired competencies based on evidence obtained during tutorial sessions. Some tutors tend to ask open-ended questions which, since they are not very specic, provide little opportunity for accurately determining students' individual levels of understanding or to make comparisons.4 Some instruments evaluate group learning skills, knowledge and critical thinking;5 others refer to responsibility, information acquisition, communication, critical sense and self-assessment. The extensive reviews of PBL carried out by Albanese & Mitchell6 and Vernon & Blake7 do not include studies related to the assessment of student performance in tutorial sessions. Des Marchais & Vu8 assessed tutor rating instruments for evaluating students' communication, reasoning and group interaction skills, autonomy and humanism. They concluded that the validity of these types of instruments is still psychometrically questionable. With the exception of Hebert & Bravo's Tutotest (1996) no other standardized instruments were found that are designed to assess medical students' attitudes and skills acquired as a result of participating in PBL tutorials. Through factor analysis of their 44-item questionnaire, the authors identied four factors: effectiveness within the group, communication and leadership skills, scientic curiosity and respect for peers. Despite the relatively low response (675%) and tutor participation (70%) rates, and variation in the number of times that some of the students were evaluated, the test was found to be a reliable and practical instrument, useful for evaluating students' skills and attitudes during tutorials. The aim of this study was to develop an instrument to assess student performance as a result of participation in PBL tutorial sessions, and evaluate its construct validity and reliability. It is part of a project that

includes the development of instruments and methods to evaluate PBL outcomes at UNAM's Faculty of Medicine.

Methods
A team of medical teachers, psychologists and education specialists was established to develop an instrument enabling tutors to assess students' attitudes and competencies acquired through their participation in PBL tutorial sessions. This team included physicians trained in PBL educational strategies at the Schools of Medicine at McMaster University, Michigan State University and the University of Limburg, Maastricht. These physicians trained and supervised the rest of the team as well as other teachers applying PBL. The behaviours to be analysed were closely linked to those involved in PBL2,911 and the UNAM faculty's experience in PBL. Such behaviours were divided into three categories: Independent study, Group interaction skills and Reasoning skills. These categories were used as frame of reference for selecting the items to be included in the questionnaire. Certain criteria had to be met: all items should describe behaviours related to PBL, be observable by the tutor during the sessions, be stated in positive terms and unambiguously, and clearly belong to only one category. In keeping with this procedure, 39 items were written and their face validity analysed by a group of 12 teachers who were asked to assign each item to a category according to its content. Of these, 21 were accepted by at least 11 teachers, eight had to be rewritten and 10 were eliminated because they were apparently not relevant or their content had already been included in other items. The degree of clarity of the 29 remaining items was rated on a ve-point scale by ve independent referees familiar with the PBL method. As a result, minor corrections were made in six of the items. Lastly, these 29 items were included in the questionnaire in random order. They describe behaviours that the tutor has to rate on a six-point scale ranging from never (1) to always (6) to indicate current frequency. The tutors who participated in this study were teachers at the Faculty of Medicine who had participated in training sessions on PBL fundamentals, strategies and evaluation. In addition, they had been using this method of instruction for the previous two years and were trained in the use of this instrument. Assessments were performed at the end of the second tutorial session. An exploratory factor analysis was carried out to identify the underlying aspects. A principal component analysis was followed by an Oblimin rotation. This

Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:818822

820

Assessment of student performance in PBL tutorial sessions

R Valle et al.

procedure was chosen on the assumption that such components would not be independent.12,13

Results
Fourteen tutors evaluated 152 rst- and second-year students in 16 tutorial groups. The sample was made up of 483% men and 517% women with a mean age of 19 years. All students attended PBL tutorials for at least 12 weeks before the study and agreed to participate in the study. The questionnaire was given once after each second tutorial group session and all 14 tutors returned the questionnaires. However, given that only fully completed questionnaires could be included in the analysis, 12 had to be eliminated, meaning that 140 questionnaires remained for analysis (Table 1). The initial factor analysis of the 29 item scale showed that two items had low load values (< 040) while three

loaded substantially on more than one factor (< 045). These ve items were therefore eliminated. Factor analysis of the remaining 24-item pool yielded 4 factors 3 with eigenvalues < 10 which together accounted for 766% of the variance, with each one accounting for 539%, 114%, 59% and 54%, respectively (Table l). On the basis of their content, factors were designated as Independent study, Group interaction, Reasoning skills and Active participation. Their Cronbach reliability coefcients were 095, 083, 094 and 093, respectively, and the total scale reliability was 096. The rst factor, Independent study, comprises nine items linked to students' initiative, motivation and persistence in searching for information, studying and achieving the learning objectives and tasks agreed on by the group. Inter-item correlations of this subscale ranged from 057 to 091. The ve items referring to factor 2 (Group interaction) concern students' abilities to function in a group, such as openness to suggestions

7 Table 1 Factor analysis of students performance in PBL tutorial sessions questionnaire* Factor loads Factor 1 Independent study 0968 0836 0824 0812 0808 0775 0717 0706 0653 A 0132 0056 0101 A 0066 0146 A 0055 0025 0148 0180 0209 0299 0105 0200 0197 0171 1293 539 Factor 2 Group interaction A 0004 A 0048 0128 A 0120 A 0124 0062 A 0160 0067 0195 0813 0777 0761 0755 0689 0146 A 0017 0043 A 0087 0022 A 0021 0253 0141 A 0029 0136 274 114 Factor 3 Reasoning skills 0103 0011 0145 0138 0107 0074 0215 0113 0092 Factor 4 Active participation 0012 A 0096 0298 A 0132 A 0134 A 0161 A 0158 A 0193 0012 A 0306 A 0023 0217 A 0015 A 0094 A A A A A A A A A 0187 0198 0055 0285 0307 0316 0741 0740 0639 0627 129 54

Items 25. Uses different resources to obtain needed information 26. Uses additional references to those suggested by the programme 21. Presents well-organized information relevant to the case 28. Is persistent in the study of the case 27. Is motivated to know more 24. Implements activities to achieve the learning objectives 29. Shows initiative in the study of the case 23. Shows initiative in the search of information 9. Accomplishes tasks agreed by the group 5. Accepts suggestions about his/her work 2. Adjusts to different group roles 1. Shows respect to his/her peers 11. Accepts decisions made by the group 6. Listens attentively to other members of the group 19. 16. 20. 15. 13. 14. 7. 3. 4. 8. Identies his/her learning needs Asks questions related to the case Accepts or rejects hypothesis on well-based foundations Analyses different components of the case Claries facts, concepts and terminology Identies information relevant to the case Gives feedback (reections, ideas and suggestions) to the group Helps her/his peers to clarify ideas Participates in case discussions Shares knowledge with the group

A A A A A A

0086 A 0175 A 0130 A 0119 0157 A A A A A A A A A A 0882 0812 0769 0669 0578 0498 0073 0027 0275 0244 142 59

Eigenvalues Contribution to variance (%) * Data derive from observations on 140 students made by 14 staff tutors.

Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:818822

Assessment of student performance in PBL tutorial sessions Table 2 Factors correlation matrix Factors Items Factor Factor Factor Factor 1: Independent study 2: Group interaction 3: Reasoning abilities 4 Active participation 1 2 3

R Valle et al.

821

0343* 0767* 0734*

0348* 0430*

0739*

n = 140. * p < 0000, one-tail test.

and decisions, adjustment to different group roles and respect towards their peers. Inter-item correlations range from 057 to 066. The six items included in factor 3 (Reasoning skills) are related to students' ability to analyse cases, formulate hypotheses, and clarify concepts. The inter-item correlations ranged from 065 to 088. The last factor was called Active participation because the four items included reect a specic form of interacting with the group that includes behaviours such as contributing, helping and sharing reections, ideas and knowledge. Here inter-item correlations ranged from 069 to 083. Pearson product-moment correlations were computed to assess the interrelations between the four factors (Table 2). Although all correlations were signicant (p < 0000, one-tail test), the highest were found between Reasoning skills and Independent study, and between Active participation and Reasoning skills and Independent study.

Discussion
The implementation of educational innovations in the curriculum, such as problem-based learning, makes it essential to evaluate processes and outcomes and to develop the necessary instruments to achieve this goal. As part of these evaluation objectives, it was decided to elaborate and assess a standardized questionnaire that could be used by PBL tutors to monitor students' progress in the achievement of PBL objectives. Results of the analysis of the questionnaire yielded four factors reecting essential components of problembased learning: Independent study, Group interaction, Reasoning skills, and Active participation.2,911 Identication of these aspects enables tutors to assess PBL components separately or as a whole, as required. They may nd it useful for assessing their students during the process (formative evaluation) or at the end

of the course or period (summative evaluation), or for evaluating the extent to which changes in attitudes and target skills are being achieved through this educational strategy. The rst factor (Independent study) reveals an underlying strategy crucial to PBL; the training of students to manage their own learning. Two of the items (9 and 21) included in this factor had originally been assigned to the Group interaction and Reasoning skills categories, respectively. However, the content of these two items would seem to be more the result of Independent study than of the other factors. The second factor (Group interaction) permits the identication of fundamental attitudes for team work, another essential component of PBL. The ability to analyse problems and propose or reject well-founded hypotheses is another fundamental aim of PBL expressed in the items comprising Reasoning skills. Finally, it is worth pointing out the distinction made between two factors: Group interaction and Active participation in the group suggest attitudes of co-operation and collaboration in the process of analysing the problem. The items included in the Group interaction and Active participation factors were originally conceived as part of a single category, but results have shown that two different aspects are involved. At the same time, the fact that the tutor is in a position to choose the type of group interaction he wishes to encourage will allow him to foster collaboration instead of competition, which is in keeping with the policies of both McMaster University14,15 and UNAM'S Medicine Faculty, which seek to encourage team work among students. On the other hand, it has been noted that the tutor should not intervene in his students evaluation since this may alter the social structure of a tutorial group.16 However, to determine the nature and extent of this inuence it would be necessary to carry out more research on the subject. The results of the correlation analysis show that the factors are signicantly interrelated, particularly Independent study, Reasoning skills and Active participation. Conversely, there is only slight correlation between Group interaction and Independent study, Reasoning skills and Active participation. These two sets are linked to intellectual activities and socio-affective activities, respectively, both crucial to the achievement of PBL objectives. The results obtained conrm the construct validity of the questionnaire and the reliability (internal consistency) of each factor. Unlike the questionnaire devised by Hebert & Bravo,17 the items comprising each factor only have a high factor load in one of them. However, this study constitutes the initial phase of the

Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:818822

822

Assessment of student performance in PBL tutorial sessions

R Valle et al. 6 Albanese M, Mitchell S. Problem-based learning: a review of literature on its outcomes and implementation issues. Acad Med 1993;68:5281. 7 Vernon D, Blake R. Does problem-based learning work? A meta-analysis of evaluative research. Acad Med 1993;68:55063. 8 Des Marchais JE, Vu NV. Developing and evaluating the student assessment system in the pre-clinical problem-based curriculum at Sherbrooke. Acad Med 1996;71:27483. 9 Barrows HS, Tamblyn RM. Problem-Based Learning: an Approach to Medical Education. New York: Springer, 1980. 10 Barrows HS. How to Design a Problem-Based Curriculum for the Pre-Clinical Years. New York: Springer, 1985. 11 Walton HJ, Matthews MB. Essentials of problem based learning. Med Educ 1989;23:54258. 12 Tabachnik B, Fidell S. Using Multivariate Statistics. New York: Harper & Row, 1983. 13 Floyd F, Widaman K. Factor analysis in the development and renement of clinical assessment instruments. Psychol Assessment 1995;7:28699. 14 Pallie W, Carr DH. The McMaster medical education philosophy in theory, practice and historical perspective. Med Teacher 1987;9:5971. 15 Neufeld VR, Barrows HS. The McMaster philosophy: an approach to medical education. J Med Educ 1974;49:104050. 16 Van der Vleuter C, Verwijnen M. A system for student assessment. In: C Van der Vleuter, W Wijnen. eds. Problem Based Learning: Perspectives from the Maastricht Experience. Amsterdam: CIP DATA Koninklijke Bibliotheek, Den Haag, 5 pp 2749, 1990. 17 Hebert R, Bravo G. Development and validation of an evaluation instrument for medical students in tutorials. Acad Med 1996;71:48894. Received 21 July 1997; editorial comments to authors 28 October 6 1997; accepted for publication 22 January 1999

psychometric evaluation of the questionnaire. Further studies are indicated not only to corroborate the results obtained with the questionnaire but also to determine the link between PBL and students' performance in the acquisition of knowledge and other skills required for physicians' training. Lastly, the questionnaire may be used for the following purposes: to track the evolution of attitudes and skills during tutorials, to serve as a tool for providing useful feedback for students and to evaluate overall tutorial group performance at the end of the course.

Acknowledgements
We would like to thank Dr Enrique Pina for his support to this project.

References
1 De la Fuente JR, Pina-Garza E, Gutierrez-Avila JH. La formacion del medico del siglo XXI y el Plan Unico de Estudios de la Facultad de Medicina de la Universidad Nac n dica Y Salud ional Autonoma de Mexico. Educacio Me 1994;28:33140. 2 Schmidt HG. Problem-based learning: rationale and description. Med Educ 1983;17:116. 3 Hay J. Tutorial performance. In: Evaluation Methods: a Resource Handbook (eds G Norman, S Shannon.). Hamilton Ontario: McMaster University, 1995. 4 Graesser A, Person N. Questions asked during tutoring. Am Educational Res J 1994;31:10437. 5 Woodward CA. Some reections on evaluation of outcomes of innovative medical education programmes during the practice period. Ann Community-Oriented Educ 1992;5:18191. 4

Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:818822