Does CRM Training Improve Teamwork Skills in the Cockpit?

: Two Evaluation Studies

Eduardo Salas and Jennifer E. Fowlkes, University of Central Florida, Orlando, Florida, and Rene J. Stout, Dana M. Milanovich, and Carolyn Prince, Naval Air Warfare Center Training Systems Division, Orlando, Florida
Crew resource management (CRM) has been the most widely used instructional strategy to enhance teamwork skills within aviation. However, few studies have provided empirical evidence that this strategy results in increased use of teamwork behaviors. The current paper describes 2 evaluation studies of CRM training that were conducted with 96 naval aviators and followed a multiple assessment approach. Results indicated that highly experienced aviators benefited from the training, which focused on increasing the use of several critical teamwork skills. These rare evaluation data suggest that CRM training is a viable strategy for improving teamwork in the cockpit. Implications for team training are discussed.

INTRODUCTION For many years the military has known that teams are essential to mission success. Military teams have been assigned to complete a wide variety of tasks, many of which are too complex or dangerous for a single individual to accomplish (Salas, Cannon-Bowers, Payne, & Smith-Jentsch, 1998). These include surveillance, damage control, command and control, and tactical aviation tasks. Teams are often better equipped to accomplish such assignments because each member possesses the distinct skills and diverse experiences needed for the team to complete its tasks successfully (Katzenbach & Smith, 1993). For similar reasons, teams are important in other specialized, nonmilitary environments such as aviation, the nuclear power industry, medical operations, manufacturing, and law enforcement. Although teamwork is necessary in the dynamic settings mentioned previously, successful team performance does not just happen. For teams to become truly effective, they must receive training in teamwork behaviors (i.e., knowledge, skills, and attitudes [KSAs] needed

to maintain effective work relations with one another) as well as task work behaviors (i.e., KSAs needed to effectively accomplish ones responsibilities; McIntyre & Salas, 1995). Many teams find it difficult to engage in effective teamwork, especially in periods of high workload. In these instances, teams tend to focus primarily on task work demands rather than teamwork, and as a result, teamwork often breaks down (e.g., Kleinman & Serfaty, 1989; Orasanu, 1990). Research in the team training arena has identified several instructional strategies that can be used to enhance the development of teamwork and task work behaviors. One such strategy is cockpit or crew resource management (CRM). Although this is perhaps the most commonly used instructional strategy for training aviation teams, there has been little empirical evidence to support its efficacy for training critical team competencies. This lack of empirical evidence has led some to question the effectiveness of this particular strategy (e.g., Besco, 1995, 1997). Our purpose in this paper is to describe a systematic evaluation of CRM training for teams that work in the highly dynamic environment of

Address correspondence to Eduardo Salas, Department of Psychology, University of Central Florida, P.O. Box 160000, Orlando, FL 32816. HUMAN FACTORS, Vol. 41, No. 2, June 1999, pp. 326343.

TRAINING TEAMWORK SKILLS naval aviation. (Although the military refers to CRM training as aircrew coordination training, or ACT, we will use the term CRM throughout this paper.) We first identify the theoretical drivers used to conduct the current training and evaluation effort. Next, we address some concerns with team training evaluations and pose two questions that we feel need further investigation. We then present two separate evaluation studies of CRM training for naval aviators in two different helicopter communities. Finally, we discuss the results of these evaluations and their overall implications for team training. Theoretical Drivers The current training and evaluation effort was guided by two theoretical frameworks: the teamwork skills identified by Prince and Salas (1993) and the team competency framework proposed by Cannon-Bowers, Tannenbaum, Salas, and Volpe (1995). Both frameworks present a set of behavioral skills that are critical to effective team performance and thus should be emphasized during training. To determine the teamwork skills needed by naval aviation teams, Prince and Salas (1993) incorporated information from three different sources (i.e., a literature review, critical incident interviews with aviators, and a team task inventory form on which aviators rated team process behaviors for importance to training and importance to mission accomplishment and safety). Seven skills emerged from the data: communication, decision making, leadership, situation awareness, mission analysis, assertiveness, and adaptability/flexibility (see Prince & Salas for a detailed description of these skills). The importance of the skills described in Prince and Salas (1993) has been confirmed through a number of efforts. These include the analyses of Navy and Marine Corps mishaps (Hartel, Smith, & Prince, 1991) and additional interviews conducted in a wide variety of aviation communities (Stout, Prince, Bergondy, Baker, & Salas, 1992). Support has been found in other domains as well. For instance, in deriving core team competencies, Cannon-Bowers et al. (1995) sorted through the variety of teamwork skills that have been offered in the literature. They identified 130

327 different skill labels, which they classified into eight major skill dimensions: adaptability, shared situational awareness, performance monitoring/ feedback, leadership/team management, interpersonal relations, coordination, communication, and decision making. These skills appear to be highly similar to the dimensions offered by Prince and Salas, supporting the argument that teamwork skills are generalizable. Training Evaluation Although tremendous strides have been made in team training research, it is still a relatively young field. Few training evaluations have been conducted to assess how and why a particular training strategy such as CRM is effective or ineffective in producing a positive change in behavior (Cannon-Bowers et al., 1989; Leedom & Simon, 1995). Reasons for this include the cost associated with performing training effectiveness evaluations and the operational constraints, both in the military and in commercial aviation, that make a systematic evaluation effort extremely difficult to perform. Fortunately, data are beginning to accumulate in this regard. Specifically, recent studies (e.g., Fowlkes, Lane, Salas, Franz, & Oser, 1994; Leedom & Simon, 1995; Silverman, Spiker, Tourville, & Nullmeyer, 1997; Smith-Jentsch, Jentsch, Payne, & Salas, 1996; Smith-Jentsch, Salas, & Baker, 1996; Stout, Salas, & Fowlkes, 1997) have begun to provide evidence that behavioral-based CRM training increases the use of teamwork skills in aviation teams. However, at least two important questions remain regarding behavioralbased training. First, can aviators who already have experience working at their tasks as a team be trained to improve their teamwork behaviors? Second, can aviators who occupy different team positions benefit from the same training program? For instance, pilots and aircrewmen (team members who do not pilot the aircraft) have access to different information, yet, at times, they must coordinate their activities very closely. We sought to answer the preceding questions in our evaluation effort. In addition, we used the theoretical drivers described previously to guide the design and delivery of the CRM courses. Our drivers were

328 fundamental in making two important determinations. First, they helped us to identify the skill areas that should be targeted during training, and second, they guided us in selecting the most appropriate instructional strategy to employ when training teams on these critical skills. Finally, to determine whether training was effective, we collected measures based on Kirkpatricks (1976) hierarchy (i.e., reactions, attitudes, knowledge, behavior). Our data collection efforts and their associated results are described in the following pages. STUDY 1 Method Participants. The participants consisted of 35 pilots (32 men and 3 women) and 34 enlisted aircrewmen (all men) from Navy transport helicopter (i.e., H-46) squadrons. They participated in the research as part of their annual CRM training course. The average age was 27.7 years (SD = 4.5) for the pilots and 24.2 years (SD = 4.1) for the aircrewmen. The mean number of flight hours in the H-46 was 690.36 (SD = 544.50) and 809.23 (SD = 682.05) for the pilots and aircrewmen, respectively. Of the participants, 3 indicated that they were instructor pilots. In addition, 32 participants reported having previous exposure to some form of CRM training (26 pilots and 6 aircrewmen). However, the course that they had been exposed to was designed to change attitudes and did not address the skills or behaviors included in the current program. Training. The content of the CRM training was driven by the seven skill areas and associated behaviors determined to be critical for good aircrew coordination in military helicopter communities (Prince & Salas, 1993). Because of time constraints and the needs of this particular helicopter community, only four of the seven skill areas were taught, in addition to an introduction module. The four skill areas were communication, assertiveness, mission analysis, and situational awareness. The training included information about CRM principles via lecture and mishap analysis, demonstration of effective and ineffective utilization of CRM principles provided through videotapes, and practice and feedback admin-

June 1999 Human Factors istered through a classroom role-play exercise. To increase participants use of the teamwork skills presented in the CRM course, each training component emphasized the skills described previously. Although the training was behaviorally focused, we also expected that participants would improve their attitudes toward teamwork and increase their knowledge of the teamwork skills as by-products of the CRM course. Measurement approach. We used a multimeasurement approach to evaluate the effectiveness of the CRM training delivered. Based on the hierarchy presented by Kirkpatrick (1976) and expanded by Cannon-Bowers et al. (1989), we collected evaluation measures on four levels. Specifically, we assessed attitudes toward the use of teamwork skills in the cockpit, reactions to the CRM course, knowledge of teamwork principles, and the use of skill competencies during a simulated flight. The measurement instruments used to collect data on each of the four levels of evaluation are described in the following sections. Trainee reactions questionnaire. The trainee reactions questionnaire included 5-point Likerttype ratings (5 = essential) and free-response items to obtain the trainees opinions on the usefulness of the training. The Likert items required participants to rate the usefulness of the training for mission accomplishment and flight. The free-response items required the participants to list ways in which they planned to use the information presented in the CRM training course. Aircrew Coordination Attitudes Questionnaire (ACAQ). The 41-item attitude questionnaire used for the current effort was largely an augmentation of existing scales. The majority of the items included were taken from the 25-item Cockpit Management Attitudes Questionnaire (CMAQ) developed by Helmreich and his colleagues (Helmreich, 1984; Helmreich, Wilhelm, & Gregorich, 1988). The CMAQ measures attitudes related to three major factors: (a) communication and coordination, (b) command responsibility, and (c) recognition of stressor effects. Items were also obtained from the Army Aviation Crewmember Questionnaire (AACQ), an adaptation of the CMAQ for Army aviation (Dynamics Research

TRAINING TEAMWORK SKILLS Corporation, 1990). Finally, additional items were included to enhance the relevance of the ACAQ to the concepts taught in the current training. Participants responded to all items using a 5-point Likert-type rating scale (5 = strongly agree). Multiple-choice knowledge test. A 17-item multiple-choice knowledge test was developed to assess the trainees recall of major concepts related to the teamwork skills taught in the course. For instance, test items required that participants correctly identify the goal of CRM training, choose the best way in which to handle conflict in the cockpit, and identify an example of feedback in the communication cycle. Behavioral assessment. An evaluation scenario and support materials were developed to evaluate each teams performance using the Target Acceptable Response to Generated Events or Tasks (TARGETs) team performance methodology (Dwyer, Fowlkes, Oser, Salas, & Lane, 1997; Fowlkes et al., 1994). In accordance with the methodology, events were included in an evaluation scenario that provided specific, observable opportunities for teams to exhibit team coordination behaviors that were targeted in the training course (i.e., communication, assertiveness, mission analysis, and situational awareness). Subject matter experts (SMEs) were involved in the selection of events that were included in the scenario and in the identification of appropriate responses to events. SMEs included two pilots (lieutenant commander and lieutenant) and one crew chief (senior chief petty officer). A large number of TARGETs (i.e., 101) were developed to enhance reliability. A TARGETs checklist was also developed that contained each mission segment, the events introduced, the associated TARGETs (i.e., acceptable responses), and space to score each TARGET as hit (i.e., executed) or missed (i.e., not executed). In addition, a scenario script was used to exert control over the delivery and timing of events to maintain task consistency across teams. The scenario developed was approximately 40 min in length. It required the team to transport troops and cargo from a small ship to an aircraft carrier located approximately 50 miles (80.47 km) away. Prior to the scenario,

329 the team participated in a preflight brief or planning period. There were 18 behavioral opportunities embedded into the preflight brief; they included reviewing the primary mission of the flight and discussing each crew members responsibilities and emergency procedures. The scenario itself was designed to contain varying levels of workload. The first segment of the flight was characterized by routine events and thus was considered the low-workload segment. A total of 42 behavioral opportunities were included in this portion of the scenario. Examples included preparing for takeoff, taking off, and completing various checklists. The second flight segment was designed to contain higher workload. During this portion of the scenario, the helicopters instruments indicated a fluctuation in transmission oil pressure caused by a filter problem that suggested an imminent transmission failure. As a result of this in-flight emergency, the team had to decide if the problem could be remedied or whether they should ditch the aircraft into the ocean. The higher-workload segment contained 41 behavioral opportunities. A portion of the significant events included in the scenario are outlined in Table 1, along with the associated TARGET(s) and skill area(s) evaluated. In addition, Table 1 emphasizes the linkages that exist among the training objectives, scenario design, and performance measurement an explicit component of the behavioral assessment approach. Because a full-mission simulator was not available for this community, the scenario was implemented on a PC-based (486 microprocessor) simulator with Microsoft Flight Simulator 4 (FS4) as the software. Several studies have shown that such tabletop simulations can be used to train and assess teamwork skills and are generally accepted by aviators (Bowers, Salas, Prince, & Brannick, 1992; Brannick, Prince, Salas, & Stout, 1998; Jentsch & Bowers, 1998; Koonce & Bramble, 1998; SmithJentsch, Salas, & Baker, 1996). Although the FS4 program does not emulate a helicopter, the SMEs were able to design a fixed-wing plane with flight characteristics that roughly approximated those of a helicopter. For example, the range of airspeeds possible and the

330

June 1999 Human Factors

TABLE 1: Sample of Nonroutine Events, TARGETs, and Skill Areas Evaluated in the Scenario (Study 1) Event Incomplete information given on cargo Ship to ship distance exceeds standard operating procedure Crew chief states he needs another minute to get troops strapped in TARGET(s) Ask questions about cargo (amount, type, whether palletized) Question whether mission should be performed Pilots check back with crew chief before taking off to see if cabin is ready for liftoff Pilots question heading information Request clarification of communication Pilots ensure that information is transferred to team members without an intercockpit communication system and that lookout doctrine is enforced Imminent transmission failure procedures executed; tasks assigned; situation status communicated; alternatives assessed; team prepared for possible ditch, etc. Decision to ditch announced, communication made to ship Skill Area Mission analysis, assertiveness Mission analysis, assertiveness, situational awareness Situational awareness

Ship provides wrong heading to carrier Garbled communication Ships air traffic control reports traffic in area

Assertiveness, situational awareness Communication Communication, leadership

Fluctuating aft transmission oil pressure

Mission analysis, leadership, communication

Intense vibration and growling noise reported from cabin

Mission analysis, communication

handling characteristics during running takeoff were somewhat similar to those of a helicopter. Joysticks were provided for flight control: one for thrust management (collective) and one for directional control (cyclic). The handling characteristics were optimized as much as possible so that the system would be easy to fly. In addition, the scenario was designed to minimize having to maneuver the

aircraft, so the scenario called for mostly straight and level flight. The system was, however, limited in that a number of routine and nonroutine flight tasks could not be evaluated. For example, there were no engine instrument gauges, and so normal, routine monitoring of fuel, engines, and hydraulics systems could not be accomplished. Despite these limitations, the SMEs were in

TRAINING TEAMWORK SKILLS agreement that enough events could be introduced to evaluate a variety of critical teamwork behaviors. That is, because the system could support a scenario that imposed realistic demands for teamwork behaviors, it provided sufficient fidelity to serve as a useful device for training teamwork. Procedure. Data were collected from a total of six CRM classes over a 5-month period. In each class an effort was made to include (a) 12 students, (b) an even mix of pilots and aircrewmen (given that each H-46 team is made up of two pilots and two aircrewmen), and (c) a range of experience levels. During the data collection period, the course was team-taught by a pilot and an aircrewman. Prior to initiation of the evaluation effort, the course had been taught several times. Therefore, over the evaluation period there was little change in course content or instructional approach. Data collection and handling procedures were established with the instructors, who were responsible for collecting all data. Additionally, two of the six classes were monitored to ensure that the intended procedures were being used by the instructors. Considering our experimental design, a combination of between-groups and within-group approaches were used to obtain the data. As indicated in Table 2, all participants received the background information questionnaire immediately prior to training, and the reactions questionnaire was administered immediately
TABLE 2: Research Design for Study 1

331 after training. The attitude questionnaire was administered to all participants on three occasions. The first two administrations were collected prior to training. Specifically, pretraining attitudes were measured from one to several days prior to the training (first pretraining assessment) and again immediately prior to the presentation of the course (second pretraining assessment). Pretraining attitudes were collected twice to investigate whether or not attitudes would improve as a result of a so-called familiarization effect, as suggested in the literature (Gregorich, Helmreich, & Wilhelm, 1990). For the third administration, the attitude questionnaire was completed by participants immediately following training, thus providing a posttraining assessment. Because of operational constraints (i.e., not all participants were available, one class of trainees followed the next), a cohort design was incorporated for the knowledge and behavioral assessment data. According to Cook and Campbell (1979), the use of cohorts is beneficial when (a) some cohorts receive a treatment but preceding or subsequent cohorts do not; (b) it can be assumed that there are only minor differences between a cohort and its contiguous cohorts; and (c) it is possible to compare the background characteristics of the cohorts that receive treatment with those that do not to determine if the samples are comparable. Cook and Campbell stressed this last point and specifically stated that

Administration Measure Background/expectations questionnaire Reactions questionnaire Aircrew Coordination Attitude Questionnaire (ACAQ) Knowledge test Knowledge test Behavioral assessment Behavioral assessment Participants All All All Pre Post

Baseline group Trained group Baseline group Trained group

332 cohort designs gain additional strength if the data analysis shows that cohort groups with and without the treatment do not systematically differ on reliably measured third variables that are believed to be possible mediators of the treatment effect (p. 127). The multiple-choice knowledge test was administered to 29 participants prior to the CRM course (students in the first, second, and sixth classes). These scores served as a baseline and were compared with scores obtained from the 38 participants in the trained group, who completed the knowledge test after attending the training course. Because of equipment limitations, behavioral assessment data were collected only for the pilots. Data were obtained from nine teams that served as the trained group (consisting of two pilots each). In addition, data were collected from five teams prior to training. This latter group served as a baseline condition. The trained and baseline groups for the behavioral assessment were not the same groups as those used for the knowledge test. Selection of the trained group was based on time and other logistical constraints, such as the availability of instructors to run the simulator. Table 3 shows flight experience for the trained and baseline groups for both the knowledge test and the behavioral assessment. The cohort design described in this study is not optimal for inferring causality. However, there are two ways to strengthen confidence in

June 1999 Human Factors the results. The first is to minimize the likelihood that the results could be attributable to mediating variables. As discussed previously, cohort designs gain strength if archival records are available to demonstrate that the cohorts do not systematically differ on third variables (Cook & Campbell, 1979). The two most critical variables that could act as potential mediators in the current study are flight experience and exposure to prior CRM training. As such, we statistically analyzed whether the baseline and trained groups for the knowledge test and the behavioral assessment differed from one another on these third variables. For the multiple-choice knowledge test, we found that the baseline pilots did not differ from the trained pilots in terms of the number of flight hours they had accumulated in the past 6 months, t(26) = 0.09, p > .05, or the total number of flight hours reported, t(25) = 0.70, p > .05, or in their previous exposure to CRM, x 2 = 0.23, p > .05. Similarly, the aircrewmen in the baseline group did not differ from those in the trained group in terms of the number of flight hours reported for the past 6 months, t(27) = 0.90, p > .05, or overall flight hours, t(27) = 0.03, p > .05, or in their previous exposure to CRM, x 2 = 0.68, p > .05. For the behavioral assessment that included pilots only, we found that the trained and baseline groups did not differ in terms of flight experience, t(19) = 1.35, p > .05 (last 6 months),

TABLE 3: Means and Standard Deviations (in Parentheses) for Age and Flight Experience Variables for the Trained and Baseline Groups (Study 1) Groups Knowledge test: Pilots Trained Baseline Knowledge test: Aircrewmen Trained Baseline Behavioral assessment Trained Baseline Age 28.5 (5.3) 27.07 (3.8) Flight Hours 773.1 (612.9) 620 (506.3) Flight Hours (Last 6 Months) 128.9 (109.1) 125.67 (87.8)

24.5 (4.6) 23.1 (2.1)

788.5 (944.5) 796.4 (240.6)

179.4 (141.1) 224.3 (120.5)

28.45 (4.7) 28.0 (5.1)

802.3 (714.0) 715.0 (391.2)

112.5 (65.5) 160.0 (120.6)

TRAINING TEAMWORK SKILLS t(18) = 0.03, p > .05 (overall), or in prior exposure to CRM, x 2 = 0.17, p > .05. Thus all analyses performed revealed no significant differences between the cohorts, increasing our confidence that cohort differences would not affect results. The second way to strengthen the results is to replicate the findings with a stronger design. We attempted to do this in Study 2. The behavioral assessment period for each team consisted of the following: familiarization with the purpose of the evaluation, a 30min familiarization exercise with the PC-based flight simulator, administration of applicable briefs, performance by the team of a teammember-to-team-member brief, and finally, performance of the mission. To minimize the spread of information about the content of the scenario, teams were not debriefed on their performances. Videotapes were obtained of each teams performance, which trained observers viewed to evaluate performance according to the TARGETs methodology. Teams were informed that they could have the videotape of their performance destroyed without penalty of any sort, although none of the teams elected to have this done. Two trained observers independently scored the tapes, spending approximately 1.5 h on each tape. Both observers were blind as to whether the teams being evaluated were in the trained or baseline group. Results Trainee reactions to CRM. Pilots and aircrewmen responded similarly to the items on the reactions questionnaire, and therefore their responses have been combined. There was strong endorsement for the usefulness of the training, with an average rating of 4.34 (SD = 0.70) out of 5. Moreover, participants considered the training to be important for mission accomplishment and flight safety, as indicated by average ratings of 4.24 (SD = 0.74) and 4.73 (SD = 0.48), respectively. A frequency analysis was performed on the responses to the question, List ways you plan to use the information presented in the course. One of the most frequently occurring responses for both pilots and aircrewmen was to improve preflight briefs (N = 13 for pilots and N = 6 for aircrewmen).

333 Attitudes. Mean scores for pilots and aircrewmen across the three test administrations for the overall attitudes score and for the CMAQ subscales are shown in Table 4. These data were analyzed using mixed-model analyses of variance (ANOVAs) with administration (i.e., first and second pretraining administrations and the posttraining administration) as the repeated-measures variable and team position (pilot vs. aircrewmen) as the betweensubjects variable. In all subsequent analyses, if a significant effect attributable to administration was obtained from the ANOVA, the difference between the pretraining administrations was assessed using a t-test. If no difference was obtained, these data were pooled and then compared with the posttraining mean. However, if a difference between the pretraining means was found indicating a familiarization or practice effect, no further comparisons were made because the meaning of differences between pretraining and posttraining means would be equivocal. On the overall ACAQ, which combined scores for all attitude scales, significant effects were obtained for team position, F(1, 42) = 8.86, p < .05, administration, F(2, 84) = 22.17, p < .05, and the Team Position Administration interaction, F(2, 84) = 3.41, p < .05. For the pilots, no difference was obtained between pretraining means. When the pretraining means were pooled and compared with the posttraining mean, a significant difference was obtained, t(20) = 6.93, p < .05. Similarly, for the aircrewmen, there was no difference between pretraining means. When the pooled pretraining mean was compared with the posttraining mean, a significant difference resulted, t(22) = 2.13, p < .05. Thus both the pilots and aircrewmen significantly improved their attitudes from the pretraining to the posttraining administration. Scores on each of the CMAQ subscales were also analyzed. First, for the Communication and Coordination subscale, the ANOVA revealed a significant effect attributable to administration, F(2, 86) = 4.10, p < .05, with no other significant main effects or interactions. Subsequent planned comparisons indicated no difference between the pretraining means. The pretraining data were then pooled

334

June 1999 Human Factors

TABLE 4: Means and Standard Deviations (in Parentheses) for the Attitudes Questionnaire Scores (Study 1) Group Overall score Pilots Aircrewmen Pooled together CMAQ Communication and Coordination subscale Pilots Aircrewmen Pooled together CMAQ Command Responsibility subscale Pilots Aircrewmen Pooled together CMAQ Recognition of Stressor Effects subscale Pilots Aircrewmen Pooled together
a

First Pretraining 3.96 (0.23) 3.80 (0.21)

Second Pretraining 3.98 (0.27) 3.82 (0.26)

Pooled Pretraining 3.97 (0.25) 3.81 (0.24)

Posttraining 4.23 (0.25)a 3.92 (0.35)a

Pooled Posttraining

4.47 (0.28) 4.40 (0.31)

4.42 (0.34) 4.40 (0.40) 4.42 (0.30)

4.67 (0.28) 4.44 (0.52)

4.55 (0.43)a

3.56 (0.88) 3.15 (0.70)

3.93 (0.64)a 3.13 (0.71)

3.14 (0.69)

4.02 (0.71) 3.51 (0.75)a

2.91 (0.68) 2.48 (0.81) 2.69 (0.77)

2.99 (0.71) 2.78 (1.00) 2.88 (0.87)a

3.22 (0.70) 2.83 (0.86)

Significant difference based on administration.

and compared with the posttraining mean, revealing a significant difference, t(44) = 2.33, p < .05. Next, for the Command Responsibility subscale, the ANOVA revealed effects attributable to team position, F(1, 43) = 8.81, p < .05, and administration, F(2, 86) = 2.63, p < .05. For the pilots, a subsequent planned comparison showed a significant difference between the pretraining administrations, t(21) = 3.3, p < .05, and no further analyses were performed. Based on the planned comparison performed for the aircrewmen, there was no difference between the pretraining administrations. However, when the pretraining means were pooled and compared to the posttraining mean, a significant difference was found, t(22) = 2.41, p < .05. Finally, the ANOVA performed on the Recognition of Stressor Effects subscale revealed that only the effect attributable to administration was significant, F(2, 86) = 6.93, p < .05. A comparison of the pretraining means indicated that they were significantly different and, therefore, no further comparisons were made. Thus significant increases in attitudes that could be attributed to the CRM course were obtained on the over-

all attitude score (i.e., the ACAQ) and on the CMAQ Communication and Coordination subscale only. Knowledge test. The mean number of knowledge test items correctly answered was 12.63 for trained pilots (N = 19) and 10.47 for trained aircrewmen (N = 19). Conversely, pilots who served as the baseline (N = 15) answered an average of 9.8 items correctly, with aircrewmen in the baseline condition (N = 14) answering an average of 8.64 items correctly. Based on the means, it appears that both pilots and aircrewmen benefited from the training. This was confirmed by a 2 2 ANOVA that was performed on the knowledge test scores with group (trained vs. baseline) and team position (pilot vs. aircrewman) as the factors. The ANOVA revealed a significant effect attributable to group, F(1, 63) = 26.7, p < .05, and team position, F(1, 63) = 13.5, p < .05; their interaction was nonsignificant. Behavioral performance evaluation: Psychometric properties. The observers agreed on 93% of the items scored. Moreover, the correlation between the overall performance scores (percentage of TARGETs hit) for the two

TRAINING TEAMWORK SKILLS

335

100 90 Proportion of TARGETs correctly managed 80 70 67 60 50 40 30 20 10 0 Pre-brief Low Workload Higher Workload 52 78 74 68 59 Trained Control

Figure 1. Proportion of TARGETs correctly managed in each scenario segment (Study 1).

observers across the 14 teams was r = .88, indicating good interobserver reliability. The performance scores across the 14 teams ranged from 58% to 79% TARGETs hit. Behavioral performance evaluation: Training effect. The overall performance scores, averaged across observers, were compared for the trained (N = 9) and baseline (N = 5) teams. Trained teams hit an average of 70% of the TARGETs, compared with 64% for the baseline teams, a difference that was statistically significant in a test for the difference between two proportions, z = 2.53, p < .05. Figure 1 shows the performance scores for each of the three segments of the scenario: the preflight brief; the low-workload segment, which was generally routine in nature; and the higher-workload segment, during which the teams were dealing with the possibility of a transmission failure. Differences were apparent, favoring the trained over the baseline teams during the preflight brief and the higherworkload segment. As indicated in Figure 1, trained teams correctly managed 15% more TARGETs than did baseline teams during the preflight brief and 9% more TARGETs during the higher-workload segment. These mission segments were characterized by a relative preponderance of TARGETs assessing behaviors from the situational awareness, assertiveness, and leadership skill areas. Tests for the difference between two proportions indicated signif-

icant differences between group performance in the preflight brief, z = 2.55, p < .05, and the higher-workload segment, z = 2.38, p < .05, but not for the low-workload segment of the scenario. Discussion Based on the four levels of evaluation, we obtained preliminary evidence suggesting that CRM training was effective in enhancing the targeted competencies (i.e., assertiveness, communication, situational awareness, and mission analysis). This evidence was strongest for the training effect on crew-member attitudes and knowledge. Specifically, the trained group showed more positive attitudes toward the use of teamwork in the cockpit and greater knowledge of teamwork principles (i.e., participants learned the teamwork principles emphasized during training). Although trained teams outperformed baseline teams during the behavioral evaluation, the results can be suggestive only of training effectiveness, given that behavioral data were not obtained for trained teams prior to receiving the training. In sum, this study begins to suggest that CRM may be effective for training welldefined teamwork behaviors. There were two problems with the study that prevent stronger conclusions. The first and more critical problem was the inability to ensure that crew members were equal prior to the training.

336 Although important demographic variables showed no obvious or statistically significant differences in what would be covariates for behavioral performance, random assignment to conditions was not possible, nor was a pretest on behavioral skills. Although this is acceptable, it is not optimal. The second problem was the nonavailability of a highfidelity simulator. Although the trainer used in the study has proved useful in eliciting teamwork skills (e.g., Jentsch & Bowers, 1998), it constrains activities somewhat and limits the demonstration of possible training effects. Despite the problems with the research, the results are suggestive of training effectiveness on several levels. However, a second study using a repeated measures approach was necessary. STUDY 2 For the second evaluation, CRM training was administered to aviators from another naval helicopter community who were more experienced than the aviators in the previous sample. Further, behavioral assessment data were collected both before and after the aviators completed the CRM course, thus providing a repeated-measures assessment. Finally, data on all measures were consistently obtained from a trained group and a control group. Method Participants. A total of 27 aviators from a naval helicopter community (CH-53) participated in this evaluation effort in conjunction with their annual CRM training course. The trained group consisted of 15 pilots; 12 pilots served as the control group. Trained pilots had spent an estimated 8 years in the military and had an average of 889.4 flight hours (SD = 727.2). Control pilots had spent approximately 8.9 years in the military and had accumulated an average of 1136.3 flight hours (SD = 805.0). A total of 14 participants (7 pilots in the trained group and 7 in the control group) indicated that they were instructor pilots. Of the pilots, 11 in the trained group and 9 in the control group reported receiving some form of CRM training prior to their participation in the study.

June 1999 Human Factors Training. As in Study 1, the content of the CRM training course was driven by the coordination behaviors and skill areas identified by Prince and Salas (1993) for helicopter communities. For this study, in addition to an introduction module, all seven skill areas (i.e., decision making, assertiveness, mission analysis, communication, coordination, leadership, adaptability/flexibility, and situational awareness) described by Prince and Salas were addressed in the CRM course. Consistent with Study 1, the training included lectures, mishap analysis, demonstration of effective and ineffective use of CRM principles, and practice and feedback administered through classroom role-play exercises. Measurement approach. The measurement approach adhered to in Study 1 was implemented for Study 2 as well. That is, we evaluated the training on four levels: reactions, attitudes, knowledge, and behaviors. To collect data on each of these levels, the reactions questionnaire and the ACAQ, described in the previous study, were also employed here. No changes were made to either of these questionnaires. A 25-item multiple-choice knowledge test was developed to assess whether or not the CRM course resulted in increased knowledge of teamwork principles. This test was similar to the one described in Study 1 in that it assessed participants knowledge of the major concepts included in the CRM course. However, a greater number of items were developed because the present CRM course concentrated on a greater number of teamwork skills. In terms of the behavioral assessment, two scenarios were developed so a repeated-measures design could be employed. Finally, the scenarios were implemented in a full-mission simulator. The TARGETs methodology was used to develop the two scenarios and the materials were used to evaluate each teams performance. Similar to Study 1, each teams performance was assessed during a preflight brief, a low-workload flight segment, and a higherworkload flight segment. In several studies researchers have observed large differences in how teams perform, interact, or both during the preflight brief and higher-workload segments of flight. As evidenced by Ginnett (1987, 1990) and Smith (1994), a teamwork-

TRAINING TEAMWORK SKILLS oriented preflight brief often translates into better teamwork during the flight and can increase the teams ability to effectively deal with time-critical situations when or if they arise. Conversely, a preflight brief that undermines the teams integrity can inhibit important teamwork behaviors, such as speaking up when one recognizes that an error has been made (i.e., assertiveness). Similarly, teams differ substantially in their ability to manage higher-workload conditions. For instance, Orasanu (1990) reported that teams that failed to engage in effective teamwork appeared disjointed and had difficulty keeping up with workload demands. Conversely, teams that engaged in effective teamwork were better able to coordinate their actions and accomplish their tasks, even in the face of illdefined problems and increased workload. Subject matter experts (SMEs) were involved in generating the events to be included in the scenarios and the appropriate responses to the events. Additionally, the two scenarios were designed to be as comparable as possible, and as such, both involved low-level navigation and were approximately 45 min in length. One of the scenarios consisted of low-level navigation to a landing zone to pick up troops. In this scenario the low-workload segment consisted of completing routine procedures, such as lifting off and completing in-flight checklists (62 behavioral opportunities). On the return flight (i.e., higher-workload segment), an aircraft
TABLE 5: Research Design Used for Study 2

337 engine failed, and the team was forced to make an emergency landing (50 behavioral opportunities). The second scenario consisted of lowlevel navigation to pick up an external load. Again, the low-workload segment was characterized by procedural events such as lifting off, communicating over the radio, and completing various checklists (74 behavioral opportunities). En route the team was informed that the load was larger than anticipated. The higherworkload segment occurred on the return flight. In this segment the load became unstable and the team had to abandon their approach when setting down the load (24 behavioral opportunities). Prior to performing both scenarios, the team participated in a preflight brief in which they reviewed the teams goals and each team members responsibilities (the preflight brief segment contained 25 behavioral opportunities for each scenario). Procedure. Data were collected from two CRM classes that were taught by an instructorpilot. Several dry runs were performed prior to teaching the classes in which data were collected for the present evaluation. As in Study 1, the instructor followed the data collection and handling procedures that we had established. The overall design employed for the evaluation is illustrated in Table 5. All participants received the background information questionnaire immediately prior to training, and the reactions questionnaire was administered immediately after training. The attitudes ques-

Administration Measure Background/expectations questionnaire Reactions questionnaire Aircrew Coordination Attitude Questionnaire (ACAQ) Knowledge test Knowledge test Behavioral assessment Behavioral assessment Group All Participants All Participants All Participants Pre Post

Control group Trained group Control group Trained group

338

June 1999 Human Factors

TABLE 6: Means and Standard Deviations (in Parentheses) for the Attitudes Questionnaire Scores (Study 2) Group Overall score Trained Control CMAQ Communication and Coordination subscale Trained Control CMAQ Command Responsibility subscale Trained Control CMAQ Recognition of Stressor Effects subscale Trained Control Pretraining 3.33 (0.24) 3.34 (0.17) Posttraining 3.35 (0.28) 3.36 (0.09)

4.43 (0.36) 4.49 (0.29)

4.43 (0.46) 4.35 (0.28)

3.89 (0.61) 3.56 (0.71)

4.11 (0.56) 3.95 (0.82)

2.77 (0.96) 2.85 (0.78)

2.66 (0.96) 2.80 (0.54)

tionnaire was administered to trained participants immediately prior to training and then a second time immediately after training. The attitudes questionnaire was administered to the control group of pilots on two occasions; approximately 6 h separated the first and second administration. The multiple-choice knowledge test was administered to participants in the trained and control groups after the course. Finally, behavioral assessment data were obtained from 8 teams (consisting of 2 pilots each) in the trained group and 5 teams in the control group. The trained group completed a scenario prior to and then immediately after completing the training, whereas the control group completed the prescenario and postscenario on consecutive days. The order of scenario presentation was counterbalanced across the teams in each group. The evaluation period for each team consisted of the following: familiarization with the purpose of the evaluation, administration of applicable briefs, performance by the team of a team-member-to-teammember brief, and performance of the mission. All other procedures were similar to those implemented for Study 1, except that the scenarios were performed in a full-mission simulator. Three trained observers, randomly assigned to teams such that two observers

scored each tape, independently scored the performance of each team. Observers were blind as to whether the teams being evaluated were in the trained or control group. Because the participants in this study were highly experienced pilots and the majority had reported previous exposure to some form of CRM training, we performed two analyses to determine if the trained and control groups were equivalent at the outset of the current CRM research. We found no significant between-group differences with regard to flight hours, t(25) = 0.84, p > .05, or as a result of prior CRM training exposure, x 2 = 0.15, p > .05. Results Trainee reactions to CRM training. Overall, pilots provided strong endorsement for the training. Specifically, the items that assessed the overall value of CRM training for flight safety and mission accomplishment received average ratings of 3.93 (SD = 0.88) and 4.33 (SD = 0.48) out of 5, respectively. In addition, the participants reported that they would implement the skills they learned in the course by providing a more extensive preflight brief (N = 4) and by debriefing all team members (N = 3). Attitudes. Mean attitude scores for the trained and control groups are shown in Table 6. The

TRAINING TEAMWORK SKILLS attitude data were analyzed with a 2 2 ANOVA with group (trained vs. control) and administration as the factors. None of the effects was significant, possibly because of a ceiling effect. That is, the means shown in Table 6 indicate that positive attitudes toward teamwork in the cockpit existed at the outset of the training, especially on the overall score and on the CMAQ Communication and Coordination and Command Responsibility subscales. Knowledge test. Trained pilots scored an average of 19.13 (SD = 2.12) items correct on the multiple-choice knowledge test compared with 14.42 (SD = 1.62) for control pilots; this difference was statistically significant, t(21) = 5.20, p < .05. Therefore, in comparison with the control group, the trained group attained significantly higher scores on the knowledge test. Behavioral assessment: Psychometric properties. The observers agreed on 89% of the items scored. Moreover, the correlation between the overall performance scores (percentage of TARGETs hit) for the two observers across the 13 teams was r = .97, indicating excellent interobserver reliability. The performance scores across the 13 teams ranged from 54% to 81% TARGETs hit on the pretest and from 57% to 88% TARGETs hit on the posttest. Behavioral assessment: Training effect. The overall performance scores, averaged across
100 90 Proportion of TARGETs managed correctly 80 70 60 50 40 30 20 10 0 Pre-test Overall Performance Post-test 72 69 76 70

339 observers, were compared for trained and control teams for the pretest and posttests. These data are shown in Figure 2; no difference was apparent on the pretest, and there was an improvement on the posttest for the trained teams only. These impressions were confirmed by tests for the difference of two proportions for the pretest, z = 1.26, p > .05, and posttest, z = 2.22, p < .05. In addition, the performance scores for each scenario segment were examined. Similar to the findings reported in Study 1, trained teams appeared to hit 20% more TARGETs during the preflight brief (see Figure 3) and 15% more TARGETs during the higher-workload segment (see Figure 4), as compared with control teams. However, the groups did not differ during the low-workload flight segment, in which they both improved on the posttest measure. Based on the results of a test for the difference between two proportions, trained teams performed significantly better than control teams during the preflight brief, z = 3.64, p < .05, and during the higher-workload segment, z = 5.42, p < .05. However, there was no significant difference between trained and control teams during the segment characterized by low workload. Additionally, trained teams significantly improved their performance during the preflight brief and the higher-workload segments from the pretest to posttest assess-

Trained Control

Figure 2. Proportion of TARGETs correctly managed overall (Study 2).

340
100 90 Proportion of TARGETs managed correctly 80 70 60 50 40 30 20 10 0 Pre-test Pre-brief Segment

June 1999 Human Factors

71 63 59 51 Trained Control

Post-test

Figure 3. Proportion of TARGETs correctly managed during the preflight brief on the pretest and posttest (Study 2).

ment (preflight brief, z = 4.37, p < .01; higher workload, z = 3.20, p < .05). Therefore, trained teams significantly increased their use of teamwork skills during two of the three scenario segments, and specifically during the segments that required more teamwork. Discussion Following the approach implemented for Study 1, this evaluation provides additional evidence that CRM training can be an effec100 90 Proportion of TARGETs managed correctly 80 70 60 50 40 30 20 10 0 Pre-test 66 78

tive training strategy for enhancing teamwork skills in the cockpit. The results obtained on each of the four levels of evaluation provide rare evaluation data. First, the reaction data revealed that pilots provided positive support for CRM training. That is, pilots felt that CRM was important for both flight safety and mission accomplishment and were able to cite specific ways in which they would implement what they had learned during training once they were back in

73 58 Trained Control

Post-test Higher Workload Segment

Figure 4. Proportion of TARGETs correctly managed during the higher workload flight segment on the pretest and posttest (Study 2).

TRAINING TEAMWORK SKILLS the cockpit. This further suggests that pilots viewed the training as directly relevant to their job performance. Second, the expectation that the CRM course would improve pilots attitudes toward teamwork was not met. Based on pretraining and posttraining scores from the attitude questionnaire, the pilots attitudes remained relatively unchanged. Considering the results reported in Study 1 of this paper (as well as those in Stout et al., 1997), this was an unexpected finding. A likely explanation for this finding is that the pilots in this sample already had fairly positive attitudes toward teamwork concepts at the onset of training. Third, pilots in the trained group attained significantly higher scores on the 25-item knowledge test than did pilots in the control group. Thus although training was not shown to affect the pilots attitudes toward teamwork, it did appear to increase their knowledge of teamwork principles. Fourth, and similarly, trained teams were able to apply this knowledge when provided the opportunity to do so. That is, trained teams demonstrated a greater number of teamwork skills than control teams when their behavior was evaluated over the course of a 45-min simulated mission (i.e., posttraining assessment). Consistent with Study 1, trained teams demonstrated a significantly greater number of teamwork behaviors during the preflight brief (i.e., 20% more) than did control teams on the posttest assessment. This finding is encouraging given that previous studies have demonstrated that establishing a team climate during the preflight brief can have a strong impact on the amount of teamwork that teams will engage in during flight performance (Ginnett, 1987, 1990; Smith, 1994). Further, trained teams engaged in a significantly greater number of teamwork behaviors during the higher-workload flight segment (i.e., 15% more) than did control teams. Whereas others have reported that teamwork typically deteriorates in high-workload situations (e.g., Kleinman & Serfaty, 1989), our training, which focused on general components of teamwork, appeared to be at least somewhat effective in providing teams with the skills needed to manage changes in workload. Considering the per-

341 formances of trained teams during the preflight brief and higher-workload segments collectively, it appears that trained teams established a climate for teamwork during the preflight brief and prepared for the flight better than did control teams. Moreover, their behavior during the preflight brief may have enabled them to later engage in more effective processes when the team encountered the nonroutine events in the scenario (i.e., higher workload, increased stress). This finding is consistent with the results reported on the effects of team planning on performance (Orasanu, 1990; Stout, CannonBowers, Salas, & Milanovich, 1998). Perhaps more important, trained teams not only outperformed control teams on the posttest assessment but significantly improved their team processes during the preflight brief and higherworkload segment from the pretest to posttest assessment, suggesting a training effect. Although the design of Study 2 was an improvement over Study 1, several limitations should be noted. For instance, sporadic data were collected for some of the participants because of data mortality. This caused a portion of the analyses to be performed on a very small number of cases. However, even given these limitations, we feel that a positive picture emerges that supports the effectiveness of CRM training. Specifically, highly experienced aviators who function as either pilots or aircrewmen participated in and benefited from our team training effort. Also, a large number of team competencies were targeted during the training, which trainees were successful in demonstrating during highly realistic, simulated scenarios. CONCLUSION Two questions were addressed. Because the studies included experienced aviators in different crew positions, we have shown that (a) CRM training can affect the teamwork competencies of those who are experienced in working in teams, and (b) the same training program can be effective for those in dissimilar crew positions. Taken together, the two studies presented in this article provide support for team training that concentrates on the critical skill competencies needed for effective performance.

342 Although some may criticize the effect sizes reported here, we should stress that Salas and his colleagues have consistently found that CRM training can improve performance by 6%20% (e.g., Fowlkes et al., 1994; SmithJentsch, Salas, & Baker, 1996; Stout et al., 1997). Such results become particularly significant when one considers the number of aviation mishaps that involve military teams. For instance, Merket, Bergondy, and Salas (in press) identified 141 Class A mishaps between 1991 and 1995 in which military aviation teams made errors related to the specific teamwork skills targeted during our training evaluation efforts. Although crew behaviors in operational environments are affected by a host of factors, such as the organizational culture, training programs targeting teamwork skills are likely to be an important element in preventing such mishaps. We feel that the results of the current evaluations are applicable to a variety of team contexts in which teams are comparable in size and structure to the teams used in these studies, teams perform tasks under environmental and organizational conditions that resemble those faced by aviation teams, and similar teamwork skills are required for effective performance. However, at least three key training principles must be incorporated in order to achieve results analogous to those reported here. That is, team training should be theoretically driven, should focus on the team competencies identified for a specific type of team within an organization, and should be evaluated on multiple levels to provide an accurate assessment of whether or not training is successful in strengthening the targeted team competencies. We encourage others to follow these training principles in conducting more rigorous evaluations of team training in aviation and other environments. ACKNOWLEDGMENTS The views expressed here are those of the authors and do not reflect the official position of the organizations with which they are affiliated. We would like to thank Tom Franz for his help with data collection, and Janis A. Cannon-Bowers, Steve W. J. Kozlowski, William

June 1999 Human Factors Rankin, and Scott I. Tannenbaum for their comments on earlier drafts of this paper. This work was performed while the senior author was at the Naval Air Warfare Center Training Systems Division. REFERENCES
Besco, R. O. (1995). The potential contributions and scientific responsibilities of aviation psychologists. In N. Johnston, R. Fuller, & N. McDonald (Eds.), Aviation psychology: Training and selection (pp. 141148). Brookfield, VT: Ashgate. Besco, R. O. (1997). The need for operational validation of human relations-centered CRM training assumptions: A critical review of curriculum content and training methodology assumptions in crew resource management training. In R. S. Jensen & L. Rakovan (Eds.), Proceedings of the 10th International Symposium on Aviation Psychology (pp. 536540). Columbus: Ohio State University. Bowers, C., Salas, E., Prince, C., & Brannick, M. (1992). Games teams play: A method for investigating team coordination and performance. Behavior Research Methods, Instruments, & Computers, 24, 503506. Brannick, M. T., Prince, C., Salas, C., & Stout, R. (1998). Can PCbased systems enhance teamwork in the cockpit? Manuscript submitted for publication. Cannon-Bowers, J. A., Prince, C., Salas, E., Owens, J. M., Morgan, B. B., Jr., & Gonos, G. H. (1989). Determining aircrew coordination training effectiveness. In Proceedings of the Interservice/Industry Training Simulation and Education Conference 11th Annual Meeting (pp. 128136). Arlington, VA: National Defense Industrial Association. Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. E. (1995). Defining team competencies and establishing team training requirements. In R. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (pp. 333380). San Francisco: Jossey-Bass. Cook, T. D., & Campbell, D. T. (Eds.). (1979). Quasi-experimentation: Design & analysis issues for field settings. Chicago: Rand McNally College Publishing. Dwyer, D. J., Fowlkes, J. E., Oser, R. L. Salas, E., & Lane, N. E. (1997). Team performance measurement in distributed environments: The TARGETs methodology. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement: Theory, methods, and applications (pp. 137153). Mahwah, NJ: Erlbaum. Dynamics Research Corporation. (1990). Development of measures of crew coordination (Tech. Report DAHC35-89-D-0030, Del. Order 0001). Ft. Rucker, AL: Army Research Institute. Fowlkes, J. E., Lane, N. E., Salas, E., Franz, T., & Oser, R. L. (1994). Improving the measurement of team performance: The TARGETs methodology. Military Psychology, 6, 4761. Ginnett, R. C. (1987). First encounters of the close kind: The first meetings of airline flight crews. Unpublished doctoral dissertation, Yale University, New Haven, CT. Ginnett, R. C. (1990). Airline cockpit crew. In J. R. Hackman (Ed.), Groups that work (and those that dont) (pp.427448). San Francisco: Jossey-Bass. Gregorich, S. E., Helmreich, R. L., & Wilhelm, J. A. (1990). The structure of cockpit management attitudes. Journal of Applied Psychology, 75, 682690. Hartel, C., Smith, K., & Prince, C. (1991, April). Defining aircrew coordination: Searching mishaps for meaning. Paper presented at the Sixth International Symposium on Aviation Psychology. Helmreich, R. L. (1984). Cockpit management attitudes. Human Factors, 26, 583589. Helmreich, R. L., Wilhelm, J. A., & Gregorich, S. E. (1988). Revised versions of the Cockpit Management Attitudes Questionnaire (CMAQ) and CRM seminar evaluation form (Tech. Report 88-3). Austin, TX: NASA-University of Texas.

TRAINING TEAMWORK SKILLS

Jentsch, F., & Bowers, C. (1998). Evidence for the validity of PCbased simulations in studying aircrew coordination. International Journal of Aviation Psychology, 8, 243260. Katzenbach, J. R., & Smith, D. K. (1993). The wisdom of teams: Creating the high-performance organization. Boston, MA: Harvard Business School Press. Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development (pp. 18-118-27). New York: McGraw-Hill. Kleinman, D. L., & Serfaty, D. (1989). Team performance assessment in distributed decision-making. In R. Gibson, J. Peter Kincaid, & B. Goldiez (Eds.), Proceedings from the Symposium on Interactive Networked Simulation for Training (pp. 2227). Orlando, FL: University of Central Florida, Institute for Simulation and Training. Koonce, J. M., & Bramble, W. J., Jr. (1998). Personal computerbased flight training devices. International Journal of Aviation Psychology, 8, 277292. Leedom, D. K., & Simon, R. (1995). Improving team coordination: A case for behavioral-based training. Military Psychology, 7, 109122. McIntyre, R. M., & Salas, E. (1995). Measuring and managing for team performance: Lessons from complex environments. In R. A. Guzzo, E. Salas, & Associates (Eds.), Team effectiveness and decision making in organizations (pp. 945). San Francisco: Jossey-Bass. Merket, D., Bergondy, M., & Salas, E. (in press). Making sense out of teamwork errors in military aviation environments. Transportation Human Factors. Orasanu, J. M. (1990). Shared mental models and crew decisionmaking (CSL Tech. Report No. 46). Princeton, NJ: Princeton University, Cognitive Science Laboratory. Prince, C., & Salas, E. (1993). Training research for teamwork in the military aircrew. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 337366). Orlando: Academic. Salas, E., Cannon-Bowers, J. A., Payne, S. C., & Smith-Jentsch, K. A. (1998). Teams and teamwork in the military. In C. Cronin (Ed.), Military psychology: An introduction (pp. 7188). Old Tappan, NJ: Simon & Schuster. Silverman, D. R., Spiker, V. A., Tourville, S. J., & Nullmeyer, R. T. (1997, December). Team coordination and performance during combat mission training. Paper presented at the Interservice-Industry Training, Simulation and Education Conference, Orlando, FL. Smith, K. A. (1994). Narrowing the gap between performance and potential: The effects of team climate on the transfer of training. Unpublished doctoral dissertation, University of South Florida, Tampa. Smith-Jentsch, K. A., Jentsch, F. G., Payne, S. C., & Salas, E. (1996). Can pretraining experiences explain individual

343
differences in learning? Journal of Applied Psychology, 81, 110116. Smith-Jentsch, K. A., Salas, E., & Baker, D. P. (1996). Training team performance-related assertiveness. Personnel Psychology, 49, 909936. Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1998). Planning, shared understanding and coordinated performance: An empirical link is established. Human Factors, 41, 6171. Stout, R. J., Prince, C., Bergondy, M., Baker, D., & Salas, E. (1992). A methodology for determining aircrew coordination needs. Unpublished manuscript. Stout, R. J., Salas, E., & Fowlkes, J. E. (1997). Enhancing teamwork in complex environments through team training. Group Dynamics, 1, 169182.

Eduardo Salas is a professor of psychology at the University of Central Florida. He received his Ph.D. in 1984 in industrial and organizational psychology from Old Dominion University. Jennifer E. Fowlkes is a senior research psychologist with the University of Central Florida. She holds a Ph.D. in experimental psychology from the University of Georgia, which she received in 1990. Rene J. Stout is a research psychologist who works in the Aviation Team Training Lab at NAWCTSD. She received her Ph.D. from the University of Central Florida in human factors psychology in 1994. Dana M. Milanovich is a research psychologist who works in the Advanced Surface Team Training Lab at NAWCTSD. She received her B.A. in 1994 in psychology from the University of Central Florida, where she is currently enrolled in the industrial/ organizational psychology masters program. Carolyn Prince is a research psychologist for the NAWCTSD/FAA/UCF Partnership for Aviation Team Training. She holds a Ph.D. in industrial/ organizational psychology from the University of South Florida, Tampa, which she received in 1984. Date received: January 29, 1998 Date accepted: October 23, 1998