Assessing Pilots Performance

for Promotion to Airline Captain

By

Timothy John Mavin

Bachelor of Aviation (Griffith University)
Graduate Diploma of Adult Vocational Training (Griffith University)
Master of Adult Vocational Education (Griffith University)

Thesis submitted in partial fulfilment of the requirements for the degree of Doctor of
Education for the School of Education, University of Queensland, Brisbane, Australia.

2010

Statement of Originality

I declare that the work presented in this thesis is, to the best of my knowledge and belief,
original and my own work, except as acknowledged in the text. The material in this thesis has
not been submitted, either in whole or in part, for a degree or any other award at this or any
other university.

_____________________________
Timothy John Mavin

2010 Tim Mavin

27th October 2010

Acknowledgments
There are numerous individuals whom I would like to acknowledge for their assistance and
guidance over the many years while doing this study. I would like to thank my initial
supervisor, Professor Peter Freebody, who gave direction and support during the initial stages
of my study. I cannot convey sufficient thanks to Dr Gloria DallAlba, who as my principal
supervisor and mentor encouraged me to maintain the highest standards in my research and
writing. Also, to two ladies, Mrs Pauline Fraser and Ms Andrea Kittila from Griffith University
who assisted me in the final edit of this thesis. For their support during the final edit, plus their
assistance improving my writing skills, I will be forever grateful. Lastly, I appreciate the seven
check captains who welcomed me into their homes to discuss issues relating to pilots
performance within the airline industry.

I would also like to thank the University of Queensland for its financial assistance, which
enabled me to attend a workshop at Bournemouth University, in the United Kingdom. This
workshop, with Professor Amedeo Giorgi, helped in my understanding of the complex
philosophy and methodology of phenomenology.

I am torn between thanking, and apologising to, my family. To my children, Joshua, Alicia,
Hugh and Amy, I am sorry for saying no to cricket, soccer, swims and games. I realise that the
time Dad spent at his computer has taken away some of the precious time a parent should
spend with his children. This time can never be recovered, but I hope that each of you will
benefit from the better person that I have become by rising to the challenge of this endeavour.
Finally, I hope my beautiful and patient wife, Karen, will come to know me as a better husband
because of the things I have learnt about myself. Karen, thank you dearly for the support you
have given me; your support has enabled me to complete this unfinished business of mine!

Abstract
This study was undertaken to investigate the assessment of airline pilots performance when
being considered for promotion to the position of airline captain. Within Australia, the
assessment standards set for pilots are based heavily on the technical skills associated with
flying an aircraft, and related knowledge. Although technical skills are important, there is,
however, another set of pilot skills which are implicated in around 70% of airline accidents.
This latter group of skills is termed nontechnical skills (NTS), and includes facets such as
decision making, communication, and situational awareness. Although so critical to airline
safety, these NTS are not explicitly identified in pilot performance assessments.

This study comprises an investigation of the implicit and explicit criteria used by assessors,
who are experienced airline captains, in making judgements about the performance of pilots
who are applying for promotion to airline captain. The research included in-depth interviews
with five experienced airline captains, each of whom possessed between 17 and 29 years
experience in assessing pilots. These interviews and related analyses were conducted utilising a
phenomenological methodology.

Despite the overt emphasis on technical skills in current regulatory standards, the research
demonstrated that an additional set of NTS (nontechnical skills) were being implicitly assessed.
It was found that three main skills criteria were assessed as essential for being a captain: flying
skills, situational awareness, and decision-making. There are of course other skills, such as
aviation knowledge, management of crew, and communication amongst crew, which assist
pilots in maintaining and improving their performance. This study concludes that the technical
metrics currently used by the regulatory authority do not encompass all of the criteria actually
used during assessment of airline pilots performance. The research has led to the development

of a new model for assessing pilots performance (MAPP), which is based on current practice,
and which makes explicit the NTS areas which are not currently identified in industry
standards. Industry use of such a model will ensure that pilots are being actively assessed in the
totality of skill areas that reflect current practice.

Table of Contents
TABLE OF CONTENTS ................................................................................................ I
List of Figures .......................................................................................................................................................v
List of Tables ....................................................................................................................................................... vi

CHAPTER ONE ........................................................................................................... 1

Why study assessment of pilot performance? ........................................................................................................1
1.1

Introduction ..................................................................................................................................................1

1.2

Commercial aviation in context ...................................................................................................................2

1.3

How safe is modern airline travel? ..............................................................................................................3

1.4

How is the pilot implicated in aviation safety? ............................................................................................5

1.5

Assessing non-technical skills (NTS) ..........................................................................................................6

1.6

Overview of thesis .......................................................................................................................................8

CHAPTER TWO ......................................................................................................... 10

Current Methods in Assessing Pilots Performance ............................................................................................10
2.1

Introduction ................................................................................................................................................10

2.2

The oversight of pilot standards .................................................................................................................12

2.3

Technical skills in pilot performance .........................................................................................................15

2.4

Non-technical skills in pilot performance ..................................................................................................17

2.4.1

Situational awareness (SA)...............................................................................................................18

2.4.2

Decision making and judgment ........................................................................................................21

2.4.3

Communication ................................................................................................................................25

2.4.4

Management .....................................................................................................................................28

2.4.5

Teamwork and cooperation ..............................................................................................................32

2.4.6

Summary of non-technical skills ......................................................................................................34

2.5

Assessing pilots performance ...................................................................................................................35

2.5.1

Assessment of pilots technical skills ...............................................................................................38

2.5.2

Assessment of pilots non-technical skills ........................................................................................40

2.6

Reviewing aviation assessment ..................................................................................................................43

2.7

Focus of the current study ..........................................................................................................................45

2.8

Summary ....................................................................................................................................................47

CHAPTER THREE ..................................................................................................... 49

Phenomenology as a research approach...............................................................................................................49
3.1

Introduction ................................................................................................................................................49

3.2

Researching the assessment of pilots performance ...................................................................................50

3.3

Phenomenology as a philosophy ................................................................................................................55

3.4

Phenomenology as an approach to empirical research ...............................................................................58

3.4.1

Collecting verbal data via semistructured interviews .......................................................................59

3.4.2

Transcribing verbal data ...................................................................................................................63

3.4.3

Reduction .........................................................................................................................................63

3.4.4

Reading of interview transcripts .......................................................................................................64

3.4.5

Dividing the data into parts ..............................................................................................................64

3.4.6

Ordering data into disciplinary language ..........................................................................................65

3.4.7

Expressing the structure of the phenomenon ....................................................................................65

3.5

Validity and reliability in phenomenological research ..............................................................................67

3.5.1

Validity .............................................................................................................................................67

3.5.2

Reliability .........................................................................................................................................71

3.6

Summary ....................................................................................................................................................72

CHAPTER FOUR ....................................................................................................... 74

Method ....................................................................................................................................................................74
4.1

Introduction ................................................................................................................................................74

4.2

Participant selection ...................................................................................................................................74

4.3

Ethical issues..............................................................................................................................................76

4.4

Trial interviews ..........................................................................................................................................78

ii

4.5

Data collection and analysis .......................................................................................................................78

4.5.1

Obtaining check captain descriptions via semi structured interviews ..............................................79

4.5.2

Transcription of the interviews .........................................................................................................80

4.5.3

Using reduction to bracket the researchers experience ...................................................................81

4.5.4

Initial analysis by reading of interview transcripts ...........................................................................82

4.5.5

Dividing the interview into parts ......................................................................................................82

4.5.6

Transforming the interview into disciplinary language ....................................................................82

4.5.7

Expressing the structure for assessing pilots performance ..............................................................83

4.6

Justification of findings..............................................................................................................................84

4.6.1

How validity was assured .................................................................................................................85

4.6.2

How reliable results were achieved and demonstrated .....................................................................91

4.7

Summary ....................................................................................................................................................92

CHAPTER FIVE ......................................................................................................... 93

Findings ...................................................................................................................................................................93
5.1

Introduction ................................................................................................................................................93

5.2

Summary of pilot assessment for each check captain ................................................................................94

5.2.1

Check Captain 1 (CC1) ....................................................................................................................94

5.2.2

Check Captain 2 (CC2) ....................................................................................................................94

5.2.3

Check Captain 3 (CC3) ....................................................................................................................95

5.2.4

Check Captain 4 (CC4) ....................................................................................................................95

5.2.5

Check Captain 5 (CC5) ....................................................................................................................96

5.3

Criteria used to assess pilots performance ................................................................................................96

5.4

Rationale and supporting evidence for the structure .................................................................................98

5.4.1

Situational awareness .......................................................................................................................99

5.4.2

Decision making .............................................................................................................................104

5.4.3

Flying skills ....................................................................................................................................109

5.4.4

Aviation knowledge .......................................................................................................................110

5.4.5

Management of crew activities .......................................................................................................113

5.4.6

Communication amongst crew .......................................................................................................117

iii

5.5

Summary of the findings ..........................................................................................................................120

CHAPTER SIX ......................................................................................................... 121

Discussion and Conclusion ..................................................................................................................................121
6.1

Introduction ..............................................................................................................................................121

6.2

The problem and how it was addressed ...................................................................................................122

6.3

Contributions of this study .......................................................................................................................124

6.3.1

Improved understanding about technical skills and NTS in pilot assessment ................................124

6.3.2

Development of a model for assessing pilots performance (MAPP) ............................................127

6.3.3

Potential for a wider application of the MAPP to assess performance ...........................................127

6.3.4

Methodological contribution to aviation research ..........................................................................129

6.4

Implications of this research ....................................................................................................................130

6.4.1

Future direction of pilot assessment ...............................................................................................130

6.4.2

Improved validity and reliability in pilot assessment .....................................................................132

6.4.3

Reducing the likelihood of conflict during assessment ..................................................................135

6.4.4

Assisting in the development of assessment scenarios ...................................................................137

6.4.5

Improved training in pilot assessment for check captains ..............................................................138

6.4.5

Improved methods in pilot training ................................................................................................140

6.4.6

Current policy in aviation ...............................................................................................................145

6.5

Conclusion ...............................................................................................................................................146

REFERENCES ......................................................................................................... 147

Appendix 1 Technical skills assessment forms .................................................................................................159
Appendix 2 Non-technical skills assessment .....................................................................................................163
Appendix 3 Participant information and consent forms ....................................................................................165
Appendix 4 Interview schedule .........................................................................................................................168
Appendix 5 Sample of CC3 interview and transformations ..............................................................................169
Appendix 6 Combined final transformation (TR3) of all five check captains ...................................................182

iv

List of Figures
Figure 1 Accident rate verse departures ....................................................................................... 3
Figure 2 Western-built hull losses per million departures ........................................................... 4
Figure 3 Trends in the causality of accidents ............................................................................... 5
Figure 4 Flight deck of a medium-sized commercial jet aircraft ............................................... 11
Figure 5 Model of SA................................................................................................................. 20
Figure 6 Model of decision making ........................................................................................... 23
Figure 7 One-way and two-way communication models........................................................... 27
Figure 8 The functional interaction of management processes .................................................. 31
Figure 9 Model of conflict resolution techniques ...................................................................... 33
Figure 10 Fidelity of simulation versus learning transfer .......................................................... 37
Figure 11 Epistemological theories of knowledge ..................................................................... 51
Figure 12 Illustration of analysis of one interview transcript for one check captain ................. 83
Figure 13 Combining all five check captains TR 3 to develop the final structure. .................. 84
Figure 14 Model for assessing pilots performance (MAPP) .................................................... 98
Figure 15 The NOTECHS Hierarchy of Levels ....................................................................... 126
Figure 16 Illustration of how MAPP can develop improved assessment measures for pilots . 131
Figure 17 Illustration of validity and reliability in assessment ................................................ 134

List of Tables
Table 1 Technical knowledge syllabus for commercial and airline pilot licence ...................... 16
Table 2 Advantages and disadvantages of one-way and two-way communication ................... 28
Table 3 Categories of management with the sub-functions ....................................................... 30
Table 4 Initial and promotional training .................................................................................... 39
Table 5 Recurrent training footprint........................................................................................... 39
Table 6 Categories and elements of NOTECHS ........................................................................ 41
Table 7 Table of common terms used in transcripts .................................................................. 81
Table 8 Categories of higher order procedures ........................................................................ 142

vi

Chapter One

Chapter One
Why study assessment of pilot performance?
1.1

Introduction

Millions of people each day board modern jet aircraft and travel long distances. It is becoming
more common to see parents placing their children on these aircraft to visit relatives, or for
school excursions. They do this in the belief that the men and women who fly these aircraft are
well trained for their job. Importantly, these pilots are assumed to be capable of performing this
work in normal and difficult circumstances.

In modern airlines, the aircraft are operated by both pilots and cabin crew. The pilot in charge
is called the captain and has ultimate authority for the safe operation of the aircraft. The
second in charge is called the first officer (FO). In some airline operations, due to extended
hours of flight, relief pilots are required. These pilots are usually of the rank second officer,
or cruise first officer. Cabin crew, traditionally referred to as flight-attendants, are required to
be on board by Australian law for the safety of passengers. Service delivery is a secondary duty
of cabin crew, although there are situations where cabin crew are not always required, such as
for freight operations. However, modern aircraft usually have pilots and cabin crew working
closely together, albeit carrying out different functions. Ability to complete required functions
is essential to the job. Particular scrutiny is given to pilots abilities, with a senior airline pilot
called a check captain having the important task of assessing pilots capabilities.

The uninitiated can be impressed by the teamwork, skill, and vast knowledge displayed in the
flight decks of a modern jet aircraft. The operation of a finely tuned airline crew, encompassing
1

Chapter One
pilots and cabin crew, working alongside engineers, ground crew, and air traffic control (ATC),
in all extremes of weather and terrain requires enormous skill. How then, does an examiner of
airline pilots (check captain), determine when a pilot is ready to take charge of a multimillion
dollar aircraft, with hundreds of passengers, and fly anywhere in the world?

This thesis is an investigation into the assessment of pilots' performance. It focuses on how
experienced Australian airline check captains, acting as assessors, judge the performance of
pilots who wish to be promoted to captain within an airline. By investigating pilot assessment
as a holistic endeavour, the study develops a broader view of pilot performance that is
reflective of current practice.

1.2

Commercial aviation in context

The use of aircraft for transportation of both passengers and freight has lead to the current
commercial aviation environment. Since Orville and Wilbur Wright made their inaugural flight
over 100 years ago, aviation has become an industry upon which the world now relies. Some of
the major advances over the last century have included improved flight instrumentation and
systems in aircraft. Such systems included propulsion in the form of jet engines, which were
used in commercial aviation soon after World War II (Mondey, 1977). The use of jet engines in
aircraft brought higher levels of reliability than those associated with older internal combustion
engines (Helmreich & Foushee, 1993). With increased reliability, air travel was able to expand
to where it is today.

The increase in traffic flow in aviation has generally been occurring ever since that first flight
on 17 December 1903. For example, data produced by Boeing (2002) illustrates steady
increases in traffic flow since the mid-1960s (see Figure 1). Latest reports from the
2

Chapter One
International Airline Transport Association (IATA) predict growth in aviation traffic flow to
continue for at least another 20 years (IATA, 2009). Additionally, Boeing (2004) proposes the
number of high capacity jet aircraft will double in this time. Even with the economic crisis that
began in 2008, IATA has anticipated a requirement for over 200,000 new pilots to be trained
worldwide in the next 9 years, with over 350,000 being required by 2026 (IATA, 2009, p. 10).
As can be seen from these forecasts, aviation is expected to expand for many years with regard
to aircraft numbers, traffic flow, and the need for pilots.

Figure 1 Accident rate verse departures

Modified from Boeing (2002)

1.3

How safe is modern airline travel?

A focus on safety has been at the centre of aviation expansion over many decades. To measure
safety, aviation sometimes looks at the unpleasant metric of accident rates leading to hull
damage, hull loss, injury, or loss of life. Figure 1 shows that the total number of all airplane
accidents and accidents with fatalities and/or total loss of life improved rapidly from the
early 1960s (Boeing, 2002). The improvement in safety can be seen despite the relative
3

Chapter One
increase in traffic flow, in the form of departures. Over the last decade, fatality rates have
stabilised relative to the steady decrease seen in the previous four decades, which can also be
seen in Figure 2 (IATA, 2008). In all, the aviation industry has succeeded in concurrently
expanding and making air travel safer over the past half century.

Hull Loss and Fatality Rate

per million departures

1.4
1.2

Industry Hull Loss

Rate

0.8
IATA Member Hull
Loss Rate

0.6
0.4
0.2

Fatality Rate

0
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Figure 2 Western-built hull losses per million departures

Taken from IATA (2008)

Paradoxically, in becoming so safe, the airline industry seems to have precluded itself from
further improving on this safety record. The trends illustrated in Figure 1 and Figure 2
demonstrate that safety has only slowly improved since around the mid-1980s, creating a
safety curve that is asymptotic, with a leveling off in accident rates. Amalberti (2001) estimates
that when a system reaches a mythical figure of 5 x 10-7, there is little likelihood of
improvements in safety. He estimates that, without improving on this figure, there will be
approximately one catastrophic event for every 5,000,000 that proceed to plan. The nature of
aviation safety has now placed it in the realm of the ultra safe system (Amalberti, 2001, p.
109). Improving on the current safety standard appears difficult at this time. Amalberti calls it
the paradoxes of the almost totally safe transportation systems (Amalberti, 2001, p. 109).
4

Chapter One
While there are great advantages that have come with the reduction in current accident rates, it
is increasingly difficult to identify ways to further advance aviation safety.

1.4

How is the pilot implicated in aviation safety?

Historically, pilot training has focused on flying skills and associated aircraft knowledge. Even
with the introduction of jet aircraft, this training approach has seen little change (Mavin &
Murray, 2010). These skills and knowledge, often referred to as technical skills (Flin,
O'Connor, & Crichton, 2009, p. 1), have been identified as important to the improvements seen
in aviation safety (Johnston, Rushby, & Maclean, 2000). However, the decline in aviation
accidents changed the percentage mix of pilot-versus machine-related accidents, as shown in
Figure 3. With improvements in safety over many years, pilots have been implicated more
frequently in aviation accidents (Nagel, 1988). Research into accidents over a period similar to
the introduction of the jet aircraft (1959 to 1989) identified pilot error as causal in 70% of
worldwide accidents (Helmreich & Foushee, 1993). So, as aviation safety has improved,
humans, and in many cases, the pilot, have been implicated in an increasing percentage of
aviation accidents.

accident causes

Relative proportion of

Human causes

Machine causes
causes
Time
Figure 3 Trends in the causality of accidents
Taken from Nagel (1988, p. 266)

Chapter One
Research has shown that the majority of pilot errors were not related to technical skills, but
rather, they were centred on areas such as communication and decision-making (Helmreich &
Foushee, 1993; Helmreich, Merritt, & Wilhelm, 1999). This research revealed a mismatch
between traditional training and assessment methods, and causes of accidents, thus opening up
opportunities for expanding other areas of pilot training and assessment (Helmreich &
Foushee, 1993; Salas, Burke, & Stagl, 2004). Some of these broader methods in pilot training
grew under the banner of cockpit resource management or, later, crew resource management.
The prime objective of crew resource management was to increase the timely and proficient
use of aircraft resources by operating crew (Orlady & Foushee, 1987, cited in Johnston, 1993,
p. 371). The training of crew resource management, later referred to as non-technical skills
(NTS), was embraced by airlines and remains foundational within airline training programs
today. Just as human performance became an obvious cause of airline accidents, the airline
industry rose to meet this finding with new training methods for pilots.

1.5

Assessing non-technical skills (NTS)

Despite new training programs being introduced, the aviation industry found the assessment of
NTS difficult. This was unacceptable to many regulatory authorities, and in 1996 the Joint
Aviation Authorities in Europe recommended that a study into NTS assessment be conducted
(van Avermaete, 1998). From this study came detailed assessment metrics for NTS, called
NOTECHS. The NOTECHS approach was to divide specific NTS areas, such as cooperation,
management, leadership, situational awareness and decision making, into smaller measurable
components. The measures developed within NOTECHS mirrored behavioural descriptors,
which presented the assessor with written criteria that matched the performance of the pilot.
This method of measuring performance required the assessor to judge only specific observable
behaviour, and avoided assessments of internal judgments by pilots (Flin et al., 2003b). For
6

Chapter One
example, the assessor could look for behavioural observations of leadership, such as
motivating statements made by the pilot to the crew, but would not attempt to elicit or evaluate
internal processes, such as pilot cognitions, beliefs or attitudes which may strongly influence
target behaviour. Although NOTECHS were developed to assist in measuring NTS in pilots
performance, there were still concerns about pilot assessment.

It has been identified that variations occur in the way that both technical skills and NTS are
being measured (Baker & Dismukes, 2002; Flin & Martin, 2001; Flin et al., 2003b; Flin et al.,
2005; Flin et al., 2009; Johnston et al., 2000). Equally important were investigations showing
inconsistency in pilot assessment within and between airlines (Brannick, Prince, & Salas, 2002;
Mulqueen, Baker, & Dismukes, 2002; O'Connor, Hrmann et al., 2002). Despite the fact that
measures were available to assist in pilot assessment, the evidence shows that there were
discrepancies in what was being measured, and how it was being measured. Without valid
measurement tools (i.e., those that measure what they are designed to measure), that can be
reliably used by different individuals and across different situations, and without such tools
being used consistently across the industry, it was not possible to accurately assess pilots
performance.

Such difficulties in assessing pilots performance presented challenges for the national
regulatory authorities who were responsible for determining a way forward in pilot assessment.
For these authorities, technical skills assessment had been based on well developed, and
longitudinally accepted measures (Mavin & Murray, 2010). This was contrasted with NTS
assessments, a new and evolving area of testing. In 2002, Australias Civil Aviation Safety
Authority (CASA) developed guidelines regarding training in NTS areas (CASA, 2002). In
2009, CASA introduced legislation regarding the assessment of NTS (CASA, 2009d).
However, this legislation was recently delayed, with CASA citing a lack of expertise about
7

Chapter One
NTS within the aviation industry (CASA, 2009e). Currently, the aviation industry is calling for
best practice (IATA, 2009, p. 30; Lehman, 2009, p. 5) procedures relating to training and
assessment of pilots in airline operations (Wilson, Guthrie, Salas, & Howse, 2010). At the time
of writing, the Australian airline industry recently gained access to the set of guidelines issued
by CASA; however, adherence to these rules has not been fully legislated, and there is no clear
path for guideline implementation across the industry. Though CASA is currently writing new
legislation, which aligns with international recommendations, it is expected that progress in
improving the assessment of NTS in pilots performance will have been stymied.

At present, the approach to assessing NTS is to divide the skill components into smaller
categories for performance (i.e., behavioural) measurement. To date, the industry has been
unable to develop a measure which accurately assesses pilots skills holistically, both in terms
of technical and NTS, with any degree of reliability within and between airlines. This thesis
argues that combining technical skills and NTS into a single, useable model that can be used
with all pilots and across all airlines, will provide a more effective method for assessing airline
pilots performance, and thus critically enhance passenger safety.

1.6

Overview of thesis

This thesis consists of six chapters. Chapter Two begins by reviewing the research literature on
technical skills and NTS in pilots performance within commercial aviation. The current
assessment approaches for each of these skill areas are then reviewed. A critical analysis of the
literature provides a foundation for the argument that there is an urgent need for a singular
model to assess pilots performance across the Australian airline industry.

Chapter One
Chapter Three describes the theoretical framework within which the current research study was
conducted, namely, phenomenology. It commences by reviewing the theory underpinning this
research. It then summarises phenomenology as a philosophy, and its development as a
methodology of research. The chapter then outlines the descriptive phenomenological
psychological method that was used in the current study (Giorgi & Giorgi, 2003, p. 243).
Finally, the chapter concludes by examining validity and reliability in a study of this nature.

Chapter Four describes the method used in the current study. The chapter considers participant
selection, ethical considerations that arose during this study, and trial interviews that were used
to test the effectiveness of interview questions. The chapter then describes the data collection
and analysis stages, and outlines how validity and reliability were demonstrated.

Chapter Five reports the findings of this research. It includes an overview of five interviews
that were conducted with airline check captains. This chapter then identifies the criteria that are
used in assessing pilots performance for promotion to airline captain, as well as the
relationships between each criterion. The research has led to the development of a new model
for assessing pilots performance (MAPP), which is based on current practice and is detailed in
this chapter.

Chapter Six argues that the current approach for assessing pilots, an approach that sees
technical skills and NTS as separate, can be improved by integrating them into a single model.
The chapter begins by identifying the contributions this study will make to current aviation
practice, and the research literature. It then examines the implications the research has on
aviation assessment, aviation training, and current policy. In its entirety, this chapter
summarises the current predicament in the industry, and suggests a means for improving airline
safety through enhanced pilot assessment.
9

Chapter Two

Chapter Two
Current Methods in Assessing Pilots Performance
2.1

Introduction

Chapter One argued that merging technical skills and NTS (nontechnical skills) into a single
model may afford a more useful means for assessing airline pilots performance. Chapter Two
extends and develops this argument, and explores the type of research that would need to be
conducted to achieve such a model. The chapter begins by looking into the oversight of pilot
standards around the world, which leads into the review of both technical skills and NTS in
aviation. Each skill area is scrutinised to develop a clear picture of the skill component being
described, as well as how it is assessed. Stemming from this enquiry, an approach to better
understanding ways to assess pilots performance is proposed. This approach is critiqued via a
reflection of the principles of educational assessment, and this critique is integrated throughout
the chapter. The chapter concludes by continuing the argument that the difficulties experienced
in assessing NTS are largely due to the fragmentation of the technical and NTS areas being
assessed, and the lack of a coherent industry-wide process.

Flying an aircraft within an airline is a complex task. For anyone capable of driving a car,
maneuvering an aircraft on the ground has some similarities. However, when airborne, a pilot
is required to process large amounts of information, so the aircraft can be flown within the
parameters laid down by CASA or their respective airline. This multifaceted task becomes
even more complex when an aircraft is flown in poor weather conditions. In this situation, only
information sourced from within the confines of the flight deck is available to pilots. Figure 4
depicts the flight deck of a commercial jet and the complex array of instrumentation available
10

Chapter Two
to the pilot (this particular aircraft is typical of the aircraft flown by pilots in the current study).
In regard to this aircraft, the captain would be seated on the left with first officer on right. In
this particular aircraft, primary instruments are duplicated in front of each pilot. Radio and
navigation systems can be found lower on the console between the pilots. Engines and
associated systems, are displayed on the two screens, on the forward middle panel. The
majority of systems are able to be activated and deactivated on the overhead panel. The
autopilot system is seen on the upper glare shield just below the main window. As would be
expected, an elevated level of skill is required to fly such an aircraft.

Figure 4 Flight deck of a medium-sized commercial jet aircraft

Taken from Boeing (2009).

11

Chapter Two
Some of the skills developed by pilots during a career include landing, take-off, and flight
manoeuvres such as turning, climbing, descending and dealing with emergencies. All of these
skills require associated knowledge, for example, speed, attitude (attitude is in relation the 3dimensional axis of the aircraft, not height above ground or water) and required power settings,
for an aircraft to climb at a particular rate. Such skills and knowledge have often been referred
to as the hard skills associated with a job, or the technical skills (Flin et al., 2009, p. 10).
Although these skills are important, other skills have also been identified as being significant,
such as decision making, communication and management skills (Flin et al., 2009, p. 10).
These latter skills, often referred to as soft skills, are now known in aviation as NTS. As
noted in Chapter One, NTS deficiencies now contribute to the majority of accidents in aviation
(Flin et al., 2009; Helmreich & Foushee, 1993). Below, the skills required to fly commercial jet
aircraft are reviewed, including technical skills and NTS, as well as how they are managed by
domestic and international regulatory authorities.

2.2

The oversight of pilot standards

Within aviation, pilot skills have been taught, developed and refined by years of experience
and practice (Mavin & Murray, 2010), and are standardised by a single international body. The
International Civil Aviation Organisation (ICAO) has been overseeing standards and
recommending practices in international aviation since the signing of the Chicago Convention
on 7 December 1944 (ICAO, 2009). Much of the standardisation and compliance of aircraft
around the world come from 18 different legislated sections within ICAO, referred to as
annexes. The annexes range from licensing of pilots (Annex 1), rules of the air (Annex 2),
operation of aircraft (Annex 6), to the running of aerodromes (Annex 14). Annex 1 and 6 are
the main focus of the present study. The recommended practices contained within these
annexes are adopted by individual countries, to standardise the flying skills and knowledge of
12

Chapter Two
pilots. If a country elects not to comply with a recommended practice, it must advise ICAO
(ICAO, 2009).

One particular function of ICAO is in the oversight of pilots performance in aviation. For
instance, over the last thirty years, ICAO identified improvements in safety that could be
achieved by human factors training for pilots (Taggart, 1987). As a result of these findings,
the ICAO Assembly in 1986 adopted resolution A26-9 on Fight Safety and Human Factors
(ICAO, 1998, p.iii). The objective of this resolution was to make national authorities aware of
the importance of human factors in aviation. Accordingly, ICAO actively published digests and
articles to improve the understanding of human factors issues (ICAO, 1990; Maurino, 1994,
1995, 1998; Johnston & Maurino, 1990; Svatek, 1990; Weisman, 1991). Additionally, in 1996,
ICAO released standards and recommended practices under Annex 6, recommending human
factors training for airline flight crews (Maurino, 1995). Soon after this, ICAO published the
human factors training manual, Doc 9683-AN/950. This document provided a compilation of
previous human factors publications seen as important for airline operational training (ICAO,
1998).

The last decade has seen ICAO actively involved in reviewing current research, education and
training programs in civil aviation. For example, during the period 2000-2004, ICAOs flight
safety and human factors program was focused on: (a) development of human error
countermeasures; (b) understanding cross cultural issues; and (c) enhancements in the area of
human-technology interfaces. These focus areas were to underpin: collection and analysis of
safety data, which included the development of safety markers; prevention of controlled flight
into terrain (CFIT); and the deployment of communication, navigation and surveillance
systems for air traffic management (ICAO, 1999). There was a change in ICAOs strategy in
the 2005-2009 years, which incorporated supporting: analysis and gathering of data;
13

Chapter Two
development of countermeasures to human performance vulnerabilities in the flight deck;
human-technology interfaces; and cabin crew safety (ICAO, 2004, p. 4). These were to assist
activities such as avoiding controlled-flight-into-terrain, preventing loss of control, and
aerodrome safety.

Although ICAO actively reviewed human factors, it could be argued that NTS have seen
reduced attention. For example, human factors include NTS, plus wider scientific disciplines
such as crew station design, fatigue, nutrition, and the use of automation in the flight deck.
However, the terms human factors and NTS are sometimes used interchangeably, requiring
national regulators, not ICAO, to define the difference (CASA, 2010). Additionally, the current
human factors training manual, Doc 9683-AN/950, has only had two small amendments since
its implementation over a decade ago, with no new NTS practices being incorporated into this
document. Although ICAO has directed resources into human factors, NTS appears to have
been neglected to some extent.

Another function of ICAO is to assist countries in development of regulations. Within

Australian aviation, regulations are governed by CASA. CASA regulates commercial aviation
under numerous federal regulations pertaining to pilot training and assessment, such as the
Civil Aviation Act 1988 (Australian Government, 1988). For instance, a high degree of
background technical knowledge is required to fly an aircraft. This technical knowledge which
must be gained depends on the aircraft being flown, and the requirements of the individual type
of pilot licence held. Like most countries, Australia has four types of licences for pilots:
student pilot licence, private pilot licence, commercial pilot licence, and airline transport pilot
licence. The student pilot licence allows an individual to undertake instruction towards a higher
category of licence, generally either the private or commercial licence. A private licence allows
a person to fly in non-commercial operations, while the commercial and airline pilot licence
14

Chapter Two
allows an individual to fly aircraft, and to be paid in commercial operations. All airline pilots
are required to hold either a commercial or airline transport pilot licence (CASA, 1988, 2004b).
However, those acting as pilot in command, or captain of an aircraft within an airline, are
required to hold an airline transport pilot licence (CASA, 2004b). With ICAO assistance,
CASA develops regulations that pertain to particular types of pilot licences.

2.3

Technical skills in pilot performance

CASA requires that pilots involved in commercial operations are tested on an extensive
amount of technical knowledge. Table 1 gives the broad areas of content to be taught and
assessed for a pilot under training for a specific licence. As can be seen, some of the content
areas are the same for the commercial and airline transport pilot licences. This is due to the fact
that as aircraft become larger, the associated systems become more complicated and require
increased levels of knowledge on the same subject that had been previously assessed. For
example, all aircraft have undercarriage (landing gear/wheels), although larger aircraft have
more complicated systems associated with the undercarriage, such as main and secondary
brakes, anti skid systems and wheel temperature probes. While assisted by ICAO, CASA
regulates the knowledge required for all civil pilots in Australia.

15

Chapter Two
Table 1 Technical knowledge syllabus for commercial and airline pilot licence
Taken from CASA (2000, 2008c)

Commercial Pilot Licence

Technical Knowledge
Aircraft General Knowledge
Flight Rules and Air Law
Radio Telephony
Aeroplane Type Knowledge
Aerodynamics
Navigation
Operation, Performance, & Flight Planning
Meteorology
Human Performance and Limitations

Airline transport Pilot Licence

Technical Knowledge
Aircraft General Knowledge
Flight Rules and Air Law
Navigation
Weight and Balance
Flight Performance and Planning
Meteorology
Human Factors

In conjunction with the technical knowledge requirements laid down in Table 1 above, CASA
requires pilots to undergo specific training and assessment on their ability to fly an aircraft in
differing situations. Initially, this would include flight training specifically orientated to skills
such as take-off, normal flight manoeuvres, landings, navigation, radio operation, and night
flying (CASA, 2008c). As pilots advance, they are required to demonstrate that they can fly
aircraft with multiple engines and with sole reference to instruments (see CAO 40.1.0, CAO
40.2.1) (CASA, 2008b, 2009a). In Appendix 1, the technical flying skills are detailed and
clearly outlined for pilots undergoing assessment. Of potential interest to the travelling public
is that pilots involved in commercial operations must undergo annual testing to maintain the
privileges of their particular licence (CASA, 2008b).

Attention to technical skills is crucial to the safety levels that aviation has achieved today
(Johnston et al., 2000). Although these technical skills are essential to aircraft safety, the sole
focus on these skills, to the exclusion of other skill sets, has been identified as problematic. As
outlined in the previous chapter, aviation has also identified NTS as significant in maintaining
safety standards. The approach taken to assessing technical skills has been the division of
flying skills into smaller components for the benefit of training and assessment. This appears to
16

Chapter Two
have served the industry well and, thus, it comes as no surprise that the development of
training and assessment methods of NTS also reflects this approach.

2.4

Non-technical skills in pilot performance

The contribution to safety of pilots working effectively together goes beyond mere technical
skills (Flin et al., 2009; Helmreich, 1995, 2000; Helmreich & Foushee, 1993). Within the field
of aviation research, differing categories of skills have been identified as important in flying.
For instance, Wickens (1999, p. 251) identified four requirements for flying an aircraft:
aviating, navigating, communicating, and system management. Yet, the first two of these
requirements do not fit into the more recent definition of NTS, which is the cognitive and
social skills that complement workers technical skills (Flin et al., 2009, p. 1). Aviating
includes controlling the aircraft within the three-axis that it flies, and using appropriate
airspeed and altitude for that phase of flight - skills normally described as technical skills, and
prescribed by current CASA regulations. Additionally, navigation requires pilots to fly a
particular path over the ground or water allowing for safeguards against terrain, weather, other
aircraft, and controlled airspace. Aviating and navigating are generally referred to as technical
skills. However, the third and fourth requirements are considered NTS. These are (a) the pilots
ability to communicate with others, such as other pilots, cabin crew or air traffic control; and
(b) the pilots ability to monitor systems related to the aircraft (system management). Also
worthy of attention are other NTS areas which have been accepted as important, such as
situational awareness (SA), decision making and teamwork (Flin et al., 2009). While aviation
has traditionally recognised that technical skills are crucial, other skills, such as situational
awareness, decision making, communication, management and teamwork, are also important to
a pilots ability to fly an aircraft, as demonstrated below.

17

Chapter Two
2.4.1 Situational awareness (SA)
SA has been identified as an important skill for pilots (CAA, 2008; de Montalk, 2008; Durso &
Gronlund, 1999; Endsley, 1988, 1994, 1998, 1999, 2010; Endsley & Robertson, 2000; Flin et
al., 2009; Hartel, Smith, & Prince, 1991; Henderson, 2009; Kern, 1997). Recognition of its
importance has evolved by the magnitude of accidents in which SA has been implicated
(Banbury, Dudfield, Hrmann, & Henning, 2002). The classical definition for SA is that
proposed by Endsley (1994, p. 31), where situational awareness is the perception of the
elements in the environment within a volume of time and space, the comprehension of their
meaning, and the projection of their status in the near future. As SA has been identified as an
important skill, the clarity of Endsleys definition has assisted researchers when investigating
concepts around perception, comprehension and projection, particularly in relation to the SA of
pilots.

Understanding the breadth of SA assists in developing a clear picture of some of the concepts
implications with regard to pilots performance. Endsley (1999, p. 260) determined that SA
includes an awareness of geographical, spatial/temporal, systems, environmental, and tactical
factors. Geographical SA, for instance, is in relation to terrain, other aircraft, and navigational
aids. Spatial or temporal SA is in relation to areas like aircraft airspeed, altitude, attitude (the
relative position of the 3 axes of the aircraft yaw, pitch and roll), flight path, and air traffic
control clearances. System SA can be referred to as an awareness of radio communications,
automatic pilot, fuel status, and flight safety parameters. Environmental SA may include
awareness of weather, icing, terrain and visibility (Endsley, 1999). Finally, tactical SA includes
an understanding of mission goals, launch capabilities, and threats to the mission. Although
tactical SA is a military concept, it could be considered, to a lesser degree, to be related to
commercial airline operations in the context of passengers, on-time performance and cabin

18

Chapter Two
crew. For example, some airlines have an emphasis on on-time-performance, requiring pilots to
be aware of how the entire flight preparation is progressing to obtain this goal.

Some of the later work by Endsley determined ways of refining SA concepts. For example,
Figure 5 shows a model of SA involving the three facets of perception, comprehension, and
projection, re-labelled as level 1, 2 or 3 SA. The perception of elements, or level 1 SA, requires
individuals to attend to a variety of sources within the environment. This may include
perception of aircraft systems, the cabin, or outside aircraft information, like terrain and
weather (Endsley, 1999, 2010). The next stage in developing SA involves comprehension of
these elements (level 2). Comprehension is more than an awareness of elements; it is an
understanding of their importance and implications with regard to the safety and efficiency of
the flight (Endsley, 1999, 2010). As Endsley (1999) outlines, when comparing novice and
expert pilots who have similar level 1 SA, an experts level 2 SA will typically consist of
higher levels of understanding than a novices level 2 SA. This is relevant because
understanding the implications of the elements in the environment aids the pilots sound
judgments.

Being able to predict future implications of the current environment is the final stage of SA
(level 3). Correctly predicting, or calculating a number of likely outcomes, assists pilots in
making informed decisions, or taking precautionary measures. For example, though a morning
weather report may signify clear weather at the destination airport, a captain may elect to carry
extra fuel, due to a particular airports propensity to fog in the early hours of the morning with
an onshore breeze. A study conducted with expert fighter pilots shows the importance of level
3 SA: Amalberti & Deblon (1992) investigated how fighter pilots developed their SA, and it
was established that expert fighter pilots spent increased time considering future scenarios,
compared to less expert fighter pilots. Spending more time generating multiple future
19

Chapter Two
scenarios, gave the expert fighter pilots greater capability than those fighter pilots who were
only generating limited and less accurate future scenarios.

FEEDBACK
WORKLOAD

Situational Awareness

Environment

Perception of
elements in
current situation

Comprehension
on current
situation

Projection of
future status

LEVEL 1

Level 2

Level 3

PRECONCEPTION
AND OBJECTIVES

DECISION

PERFORMANCE
OF ACTIONS

INDIVIDUAL
(ABILITY,
EXPERIENCE,
TRAINING)

DOCTRINE,
RULES AND
PROCEDURES

Figure 5 Model of SA
Taken from Endsley (1994, p. 31)

Research has shown that accidents caused by human factors may be largely attributable to SA
errors, but, the three levels of SA are not proportionally accounted for in SA-related accidents.
For instance, in Endsleys (1999) study, 71% of aircraft accidents could be attributed to human
error, with SA accounting for 88% of this total. Of the SA-related accidents, deficits in
perceiving the situation (level 1) accounted for 80.2% of the total 88%, failure to comprehend
the situation (level 2) contributed 16.9%, and failure to project SA (level 3) was a mere 2.9%.
To support these findings into level 1 SA being over-represented in accidents, research shows a
difference between individuals who do or do not achieve high levels of SA. The research
shows higher-achieving individuals are more likely to be more deliberate in seeking
information, and faster to detect and respond to changing situations (Endsley & Robertson,
2000). Clearly, failures in level 1 SA are over-represented in these statistics. Appreciating that
20

Chapter Two
SA is highly represented in aviation accidents, and that level 1 SA is a major reason for this,
strongly supports the importance of pilots abilities to perceive information from the
environment in which they work.

The initial work by Endsley, as well as contributions by other researchers, has been significant
in defining and clarifying the way that SA is researched and understood. Being able to
accurately measure the level to which a pilot has obtained this skill is therefore critical to any
pilot performance assessment.

2.4.2 Decision making and judgment

Decision making and judgment are common areas of research in aviation, with poor decision
making being identified as contributing to many aviation accidents in the past (Flin et al., 2009;
Wiggins, Connan, & Morris, 1996). For example, crashes like that of a British Midlands
Boeing 737-400 (aircraft registration G-OBME) in 1989, was reported by Air Accident
Investigation Branch (AAIB) as being due to the pilots accidentally shutting down the wrong
engine on a two-engine aircraft (AAIB, 1990). This accident, along with others that have
occurred in aviation, have instigated a greater focus on decision making by pilots.

With decision making being identified as an underlying reason for aviation accidents,
researchers have examined the way in which individuals make decisions in complex
environments. Some of the initial research aimed at defining decision making utilising the
classical decision making model (Klein, 1993). Proponents of the model theorised that
individuals identify and review alternatives, analyse likely outcomes in relation to consequence
and risk, and then make decisions. However, it was subsequently argued that this initial theory
did not match the reality of decision making process in humans (Beach & Lipshitz, 1993;
Klein, 1993; Orasanu & Connolly, 1993). One critic stated that by 1989, it was fairly clear
21

Chapter Two
how people didnt make decisions, and the next thing was to examine why they did not make
them in a certain way (Klein, 2008, p. 456). By the 1990s, it became clear that many accidents
could be attributed to poor decision making by pilots. Furthermore, it was realised that
understanding the human decision making process would be critical to improving the
decisional abilities of pilots of the future. However, with research still progressing, what was
required was to understand (a) how decisions were made; and (b) how pilots could be trained to
make better decisions.

The initial rejections of classical decision making theory paved the way for other theories.
Under the banner of naturalistic decision making, a new body of research developed, which
aimed to construct a more authentic model of human decision making. The researchers
concluded that, during decisions, individuals assessed a situation, projected likely
consequences of a possible single option to consider adverse outcomes, then, if none were
found, decisions were implemented (Orasanu & Connolly, 1993). The discovery was called the
recognition-primed decision model. It worked on the view that, to make decisions, individuals
observe a situation, check for any significant match from a previous store of information, and
anticipate an outcome with possible consequences (Klein, 2008). For example, a pilot may
recognise a particular weather pattern, such as a thunderstorm, and simply turn the aircraft to
avoid the storm, even though many factors were considered, such as air traffic control
boundaries, other aircraft, turbulence penetration speed, seating passengers and cabin crew; a
process that can be conducted quickly and safely. Recognition-primed decisions can be
conducted very rapidly, and explain how individuals can make quick decisions in complex
environments (Klein, 2008). When scenarios differed slightly and became unfamiliar,
individuals would select previous experiences and patterns that provided a best fit. As Klein
(2008, p. 457) states, decision making is a balance of intuition and analysis. The recognitionprimed decision model is supported by other research that suggests that individuals who have
22

Chapter Two
large memory stores from previous experience are better able to transfer previous knowledge to
new and novel situations (Billett, 2001).

Research into decision making over the last decades has enabled better models to be
developed. Flin et al. (2009) incorporated the research outlined above into a simplified model.
Figure 6 shows that, according to this model, decision making commences with a course of
action called situational assessment. Situational assessment is the process whereby
individuals use their awareness of a situation in diagnosing the situation (Flin et al., 2009). In
the case of individuals familiar with a particular event, their base of knowledge enables the
initial assessment to be rapid (Orasanu & Connolly, 1993). By analysing many different
accidents, from a variety of professions, Flin et al. (2009) highlighted that situational
assessment, rather than reasoning ability, is a foremost characteristic in differentiating novice
and expert decision making. The initial assessment, which is affected by experience, is the
beginning of the decision making process.

Figure 6 Model of decision making

Taken from Flin et al. (2009, p. 44)

23

Chapter Two
The next step in the model takes into account the perception of time constraints and risk. Time,
both actual and perceived, has been identified to affect the decision process (Billett, 2001;
Cohen, 1993; Klapproth, 2008). For instance, experts make better judgments in regard to the
time required for a decision, or have a repertoire of techniques to make time prior to deciding
on a decision (Billett, 2001). Risk also affects decision making, with increased levels of stress
from high risk settings influencing information processing (Flin et al., 2009). Once an
individual has identified the time and risks associated with a problem, they are able to make a
suitable decision.

In the model by Flin et al. (2009) in Figure 6, there are four decision options. The first is the
intuitive decision, which is based on recognition-primed decisions already outlined above
(Klein, 2008). The second option is a rule-based decision, which requires individuals to
correctly identify a situation and apply a rule or checklist to a given scenario (Flin et al., 2009).
For example, in aviation, most non-normal (i.e., emergency) operations can range from simple
systems malfunctions to life threatening conditions, requiring the pilots to identify a
malfunction/s and then, by referring to a checklist, action the items contained for the specific
malfunction/s. This is in contrast to the past practice of using their best judgment to deal with a
malfunction.

The third decision option in Flin et al. (2009) is an analytical decision. If an individual finds,
during situational assessment, that no previous situation is recognised or no rule is available, a
variety of options will be generated and considered prior to a decision being made. This
method of decision making aligns itself with the classical decision making model previously
explained (Klein, 1993). The benefits are that final judgments, made according to classical
decision making, are more likely to be the optimal decision (Flin et al., 2009). In contrast,

24

Chapter Two
recognition-primed decisions are satisficing rather then optimising (Orasanu & Connolly,
1993, p. 18).

The final decision option available in the model is that of creative decision making. When
individuals find themselves without previous experience or procedures, they are required to
consider innovative decisions (Flin et al., 2009). For instance, widely quoted within aviation
circles is the accident of a DC-10, United Airlines Flight 232. An engine failure of the centre
engine severed hydraulic lines to the aircrafts flight control system, rendering this system of
no use. The pilots, with assistance from a third pilot from the cabin, using differential thrust on
the aircraft engines, were able to land the aircraft, with only minimal loss of life (Flin et al.,
2009). In the Flin et al. (2009) model, then, after situational assessment, individuals have four
options available to them during decision making, each of which is affected by experience,
time available, and risk.

Decision making has been identified as an important NTS, and has been acknowledged to be
causal in many aviation accidents. Though its importance is recognised, decision making
theory may have confused many in aviation, with the early classical decision making models
underpinning current assessment criteria. As will be discussed later in this chapter, this
confusion may lead to reduced assessment effectiveness.

2.4.3 Communication
Communication is a complex and diverse area of NTS and is acknowledged as a vital
component in teamwork environments. Over the years, aviation texts have continued to review
accidents that have been attributed to poor communication, or communication not relaying the
intended meaning from the sender (Edkins, 2005; Foushee & Helmreich, 1988; Kanki &
Palmer, 1993). Early studies that focused on aviation accidents around the 1950s identified that
25

Chapter Two
during critical communication - important parts of the message - could be contained within a
small fraction of the message (Hawkins, 1987). This led to a greater standardisation of
phraseology used between pilots, air traffic control, and between pilots and cabin crew. Even
with early standardisation, communication was still acknowledged as critical in the completion
of specific tasks (LePlat, 1991). Communication has also been viewed as fundamental to all
areas of NTS (Flin et al., 2003b). It is considered as a vital management tool and is seen as a
key way in which crew are able to transfer information (Flin et al., 2009). Even with early
research leading to improved standardisation of communication within aviation; it is still seen
as an important skill for pilots, for example, being able to interact with other aviation
personnel.

Like all areas of NTS, communication has become a topic of research, with models being
developed to reflect the different schools of thought in the area. For instance, communication
models are described typically as a one-way or two-way process (Flin et al., 2009). One-way
communication sees information going from one person to the next. Meaning from the
individual is encoded and transmitted to the receiver who in turn decodes the message to
determine meaning (see Figure 7). Two-way communication allows the receiver to interact
with the sender, thus closing the feedback loop (Flin et al., 2009). Table 2 below presents some
of the advantages and disadvantages of two-way communication in comparison to one-way
communication.

26

Chapter Two

Transmission

Encode

Decode
SENDER

Meaning

RECEIVER

Meaning

Transmission

Encode

Decode
SENDER

Meaning

RECEIVER
Meaning

Decode

Encode
RECEIVER

SENDER

Transmission
Figure 7 One-way and two-way communication models
Taken from Flin et al. (2009, p. 71)

Even when communication is understood as a one-way or two-way process, another factor that
can influence its effectiveness is the different meaning generation that can occur when people
in multiple roles interact. For example, communication between the flight deck, cabin crew or
ground engineers: studies have shown that although individual crew members may use similar
language, they may come from different cultures or communities of practice, which may create
problems during the coding and decoding process (Chute & Weiner, 1995; Munro, Kanki, &
Jordan, 2008).

27

Chapter Two
Table 2 Advantages and disadvantages of one-way and two-way communication
Taken from Flin, et al. (2009, p. 71)

One-way Communication
Advantages
-Rapid
-Looks and sounds neat
-Sender feels in control

Two-way Communication

Disadvantages

Advantages

-Requires planning
-Responsibility for clear
message is with the
sender
-No feedback to sender

-More accurate
-Permits checking of
details
-Less planning required
-Receivers are able to
make more accurate
judgments
-Both sender and receiver
are responsible
-Sender and receiver work
together to develop
understanding

Disadvantages
-Takes longer
-Receiver has to
communicate in return

Other issues in effective communication come from the prioritisation, appropriateness, and
initiation of the flow of safety-critical information between cabin crew and pilots. Research has
identified that effective communication of cabin crew with pilots is reduced when cabin crew
show signs of self doubt, employment concern, and misconstrued or incorrect interpretations of
communication protocols (Chute & Weiner, 1995). Despite the fact that individuals working in
team environments may have standardised communication codes of behaviour, other issues
relating to differences in culture or communities of practice affect the way that communication
is initiated, prioritised, delivered and understood.

2.4.4 Management
The conceptualisation of management varies widely in aviation. The broader view sees
management as both a cognitive and social function, given that it requires cognitive processes,
but is enacted within a social environment (Flin & Martin, 2001; Wickens, 1999). However,
within NTS, and in particular the NOTECHS system of measurement, management is seen as a
social function (Flin & Martin, 2001; Flin et al., 2003b, 2005). It has been defined as
28

Chapter Two
coordinating and overseeing the work activities of others so that activities are completed
efficiently and effectively (Robbins, Bergman, Stagg, & Coulter, 2009, p. 10). There is a
paucity of research on management specifically relating to the work of pilots. While there is
ample research on teamwork, there is scant work on how pilots and, in particular, captains,
should coordinate and oversee crew activities. A small number of articles refer to automation
as a means of coordinating machine-human interactions (Olson & Sarter, 2000), although it is
argued that automation is not a management tool, but a tool that assists in reducing workload.
So whilst aviation has developed definitions of management, there is a need for more specific
research, describing best practice in management approaches for pilots, who organise and
supervise activities in airline operations.

The few articles that were identified in the area of management by pilots were from the
National Aeronautics and Space Administration in the United States some 16 years ago, which
defined four categories of management as level-1 functions in the flight deck. These categories
were flight management, communication management, systems management, and task
management (Abbott, 1993). Abbott expanded each of these categories into functions,
including level-2 and level-3 functions (see Table 3 below). Flight management was split into
two level-2 functions of flight guidance and flight control. Flight guidance includes
development of a desired plan, identifying required resources, assessing the current situation
and, if required, determination of new courses of action. Flight control refers to management of
the flight path, speed, and attitude of the aircraft. The level-1 category of communication
management includes three functions of receiving, processing, and sending information. These
functions were similar to communication outlined above (2.3.3). Systems management
incorporates functions of configuration planning, monitoring of systems (such as flaps and
fuel), assessment and, if required, action. The final level-1 function, task management, refers to

29

Chapter Two
the management of the resources required to conduct a mission. It includes the coordination
and overseeing of activities of the other level-1 functions.

Table 3 Categories of management with the sub-functions

Taken from Abbott (1993, p. 4)

Flight Management
Flight guidance
- Planning
- Monitoring
- Assessing
- Determining actions
- Modifying
Flight Control
- Planning
- Monitoring
- Assessing
- Determining Actions
- Modifying

Communication
Management
Receiving
- Monitoring
- Acquiring
- Storing
Processing
- Interpreting
- Evaluating
- Formulating
Sending

Systems
Management
Configuration planning
Monitoring
Assessing
- Comparing
- Diagnosing
Determining actions
Modifying

Task Management
Monitoring
Scheduling
Allocating

Where:
2nd Level function
- 3rd Level function

The four categories outlined in Table 3 are not independent of each other, but are integrated
into a larger, more holistic view of management (see Figure 8). Given that these categories are
interlinked, Abbott (1993) argued that this has implications for the efficiency of pilot activities,
especially when moving between categories. For example, crew may be managing the flight
guidance of the aircraft by reviewing the fuel status, and then be required to modify flight
control by diverting to another airfield to counter a low fuel problem. In this situation, a switch
occurs between tasks in the same category (flight management). However, when air traffic
control communication is included, which belongs to a different category, the management
complexity increases and is accordingly more difficult. This means that transferring within
functional management areas is easier than transferring between them (Abbott, 1993). It was
also noted by Abbott that even the most simple of tasks within the flight deck will include each
of the four management functions in some form.

30

Chapter Two

Flight
Management

Task
Management

Communication
Management

System
Management

Figure 8 The functional interaction of management processes

Taken from Abbott (1993, p. 10)

Research into measuring variations and effectiveness of each of these management functions
found that task management was a major determining factor in how effectively crew performed
outside normal, everyday operations. Work by Schutte and Trujillo (1996) found that, in nonnormal operations, crews generally spent a large amount of time in task management, including
scheduling and allocating resources. They also discovered that crews who were able to
effectively prioritise activities (a task management function), based on the severity of the
event, performed better. Another finding was that pilots who had the capacity to maintain focus
during interruptions were more effective. Surprisingly, it was found that some of the principles
that had been taught in NTS training, such as in communication, were sometimes distracting
and could be better prioritised (Schutte & Trujillo, 1996). The research outlined that task
management, the management function that monitors, schedules and allocates resources,

31

Chapter Two
became an extremely important factor in measuring the efficiency of how a captain manages an
aircraft (Abbott, 1993).

2.4.5 Teamwork and cooperation

The term teamwork has already been used in this thesis to describe how pilots and other
members of an airline work together, and it is not surprising that teamwork has been identified
as a nontechnical skill. In the early development of NOTECHS, the original focus was on
cooperation rather than teamwork; cooperation related to an individuals ability to work in a
team, rather than directly related to the job itself (Flin, Goeters, Hrmann, & Martin, 1998).
Later, the term cooperation became a subset of teamwork, even though comparable measures
to the original cooperation category were utilised (Flin et al., 2009). Teamwork has been
defined as:
groups of people embedded in organizations, performing tasks that contribute to
achieving the organizations goals. They share overall work objectives. They have the
necessary authority, autonomy, and resources to achieve these objectives. (West,
2004, p. 18)

Four main areas identified as important in teamwork are: exchange of information, coordinate activities, supporting others, and solving conflict (Flin et al., 2009, p. 94). The
crossover within NTS is readily identified with the first two areas - exchange of information
and coordination - already covered in the areas of communication and management,
respectively. The third important area in teamwork - supporting others - is vital to developing
and maintaining the interests of those in a group (West, 2004).

Solving conflict is the fourth area seen as important in teamwork. It is recognised that three
types of conflict can occur: task conflict, relationship conflict and process conflict (Robbins
32

Chapter Two
et al., 2009, p. 444). Task conflict relates to disagreements about the aims. Relationship
conflict refers to difficulties due to differences between individuals. Process conflict occurs
when team members disagree on the strategies and approaches to how a job is to be carried out
(Robbins et al., 2009).

Individuals within teams have options to solve conflict (see Figure 9), which generally are
related to levels of assertiveness and cooperation (Robbins et al., 2009). For example, airline
captains may choose varying degrees of assertiveness and/or cooperativeness, depending on the
task. Additionally, it could be suggested that a captain must act in ways that facilitate crews to
vary their assertiveness or cooperativeness levels, so that best outcomes can be achieved. For
example, a captains approach during a non-normal situation may create conflict, although the
captain must be able to judge if this conflict is in the best interest of achieving the aim of
landing the aircraft safely. Alternatively, a more relaxed, supportive approach may be used
during briefings on the ground.

Figure 9 Model of conflict resolution techniques

Taken from Robbins et al. (2009, p. 445)

33

Chapter Two
Another model of teamwork, that spills over into other areas of NTS, such as SA and decision
making, is that of shared mental model theory (Janis, Cannon-Bowers, & Salas, 1998, p. 26).
Shared mental model theory assumes that all members in a team are involved in a process by
which team members develop shared models of the teams internal and external context (Salas
et al., 2004, p. 331). In high workload and high stress environments, teams are able to maintain
a level of effective performance, even when inter-team communication has been restricted
(Janis et al., 1998, citing Kleinman & Serfaty, 1989). To do this, teams who have a shared
understanding of their goals, and the processes to achieve specific goals, are better able to
predict the likely requirements of their tasks, and those of the team (Janis et al., 1998). Citing
many accidents both in aviation and other highly skilled jobs, Flin et al. (2009, pp. 100-101)
outlined some of the reasons for teams not performing well; these included roles not clearly
defined, lack of explicit coordination, and miscommunication. Shared mental models have
been identified as important in teamwork, with recommendations by some researchers to
include specific training on their development (Salas, Rhodenizer, & Bowers, 2000; Wilson et
al., 2010). Shared mental models in teams can be an effective antidote, even when situations
become stressful or workloads suddenly increase (Orasanu, 1993). Having team members
develop shared mental models is seen to assist each team member to understand the teams
goals and responsibilities, whilst allowing the identification of any miscommunication or
misunderstanding.

2.4.6 Summary of non-technical skills

It has been acknowledged that NTS are extremely important in the overall performance of
pilots (Flin et al., 2009; Helmreich & Foushee, 1993; Helmreich et al., 1999). This section has
given an overview of NTS, highlighting SA, decision making, communication, management
and teamwork, together with current studies and approaches to NTS in aviation. To suggest
that NTS are broad and complex would be an understatement, and the swathe of information,
34

Chapter Two
approaches and concepts seems to be building. Additionally, it has been recognised that some
of the work done on NTS, for example, in the area of decision making, has not reflected
practice (Janis et al., 1998). Furthermore, there appears to be a blending of some areas of NTS,
as can be seen from the section above on teamwork.

The substantial research on NTS has improved our understanding of the centrality of these
skills to pilots work. All of this research has broadened our appreciation of the complexity of
pilots performance in flying aircraft safely. Nonetheless, it is argued that the research has
fragmented NTS to such a degree that few individuals in aviation are able to comprehend how
they all fit together. As a consequence, this fragmentation of skills may have created a more
complex system than is necessary. The result is that those involved in assessing pilots
performance have found implementing such a complex process difficult. This argument will be
continued in the next section.

2.5

Assessing pilots performance

This chapter so far has examined the technical skills and NTS required to fly commercial
aircraft. It has argued that the approach taken in investigating such skills may have created an
environment that is more complex than may be necessary. The current approaches used to
assess both technical skills and NTS will now be reviewed. Assessment tools and metrics used
to assess pilots performance will also be outlined.

A variety of assessment tools can be used to assess pilots performance, as laid down by ICAO
and adopted by CASA. In Australian civil aviation, knowledge assessment is based on
multiple-choice and short-answer questions (CASA, 2009c). During practical assessment, these
are supplemented with random questioning by assessment officers during flight tests (CASA,
35

Chapter Two
2008b). Skills assessment is conducted within an aircraft or a simulator. Simulation is a tool
that is extensively utilised in training and assessment in aviation.

Simulators play an increasingly important role in the assessment of airline pilots and, as a
training tool; they are invaluable (Lee, 2005; Mavin & Murray, 2010; Roscoe, 1980). In
todays airline industry, simulators are exact mock-ups of aircraft cockpits, enabling sensations
of acceleration/deceleration and turbulence, with graphics that assist in making the simulator as
realistic as possible. In relation to a pilots ability to learn, learning transfer is the term used
to measure this effectiveness, and varies in relation to simulator fidelity, with increasing
realism, to a point, improving learning transfer (Roscoe, 1984, p. 196). However, if simulation
becomes too realistic (refer to Figure 10), during early training, there is the likelihood that too
much information can be counterproductive (Moroney & Moroney, 1999; Roscoe, 1984). In
addition to learning transfer, for those who use simulation as a tool for training and assessment,
it becomes necessary to consider cost. Figure 10 illustrates that, as the fidelity of a simulator
increases, the cost of simulation grows exponentially. That is, for little gain in realism in a
sophisticated simulator, costs become large (Roscoe, 1984). Roscoe also illustrates (see Figure
10) that simulators that fall in the honey region provide effective platforms to delivering good
assessment and training (Roscoe, 1984, p. 197).

36

Chapter Two

Figure 10 Fidelity of simulation versus learning transfer

Taken from Roscoe (1984, p. 197)

In Australia, simulators are classified as either full flight simulators which have high levels of
realism, or flight training devices which do not fully replicate all components of aircraft
motion. To assist airlines, CASA provides guidelines on the type of simulator that can be used
during training and assessment of pilots (CASA, 2004a). Most airlines continue to use full
flight simulators for training and assessment of pilots. As an assessment tool, these simulators
enable far more scenarios to be assessed. For example, the Australian Transport Safety Bureau
(ATSB) identified that on a particular commuter aircraft, pilots were able to be given only 13
malfunctions whilst safely flying the aircraft. This is in contrast to the 225 malfunctions that
can be practised within a simulator that is replicating the same aircraft type (ATSB, 2009, p.
34). In addition to this, other aviation incidents and fatalities have been attributed to the lack of
simulator use in training and assessment (ATSB, 1995, 2000, 2006). Although simulators are
used extensively in airline training and assessment, an additional platform that is available in
training and assessment in aviation is the aircraft itself.
37

Chapter Two
Another important aspect that must be considered is in regard to those individuals who are
tasked with assessing pilots performance. Within airlines, senior airline captains involved in
training departments are generally divided into two categories: those able to train pilots and
those able to train and assess pilots. Throughout the world, there are a variety of names given
to the pilots in these positions. In Europe, they are often referred to as type rated instructors
and type rated examiners. In Australia, they are referred to as training captains and check
captains. CASA recognises three differing levels of check captains, and accordingly, as their
experience increases, they are able to take on greater responsibility (CASA, 2008a). For
example, given the importance of a captains position, a pilot being assessed for promotion to
captain would require the check captain to have a higher level of experience, whereas a new
check captain may be restricted to conducting surveillance assessments on already qualified
pilots, or assessing new first officers. Throughout this thesis, the term check captain will be
used to refer to pilots who have been approved by their company, and by CASA, to assess
airline pilots performance.

2.5.1 Assessment of pilots technical skills

Technical flying standards are a vital component in the assessment of pilots. CASA has
detailed standards for flying skills for pilots undergoing testing for issue, and renewal of
specific licences. Appendix 1 includes a list of technical flying standards for the commercial
pilots licence and instrument rating. Like the knowledge requirements, these standards are
explicit and detailed. For example, the acceptable standard for maintaining cruising altitude is
plus/minus 100ft, and tracking tolerances whilst navigating using a particular navigational aid
are plus/minus five degrees. Once a pilot joins an airline, there are additional training and
assessment requirements. Within Australia, there is no significant difference between the ways
in which different airlines induct, train, and assess pilots. Tables 4 and 5 give a broad example
of a training footprint taken from an airline currently operating within Australia. Each area
38

Chapter Two
within the footprint is assessed using a variety of tools mentioned above, and the metrics laid
out in Appendix 1.

Table 4 Initial and promotional training

Training

Initial type endorsement

Induction (FO & Captain)

Initial First Officer
Captain Upgrade
Training Captain Upgrade
Check Captain Upgrade

Methods

Classroom/computer
instruction
Simulator instruction
Classroom instruction
Simulator + Aircraft
Aircraft Line Operations
Classroom instruction
Simulator + Aircraft
Classroom instruction
On-the-job training

Assessment

CASA or delegate

Generally
on
specific
knowledge items from CASA
Assessed by check captain
Assessed by check captain
Nil
CASA

Table 5 Recurrent training footprint

Training

Instrument rating renewal

(yearly)
LOFT
(2 sessions every 6 months)
CRM
(yearly)
Emergency procedures
(yearly)

Methods

Classroom and Simulator

Simulator
Classroom
Classroom
Simulation

Assessment

Oral testing by
CASA or delegate
Check captain of approved
training pilot
NIL
Generally
on
specific
knowledge
items
from
CASA. Assessed by CASA
or delegate

39

Chapter Two
2.5.2 Assessment of pilots non-technical skills
The required technical skills assessment metrics have been refined over almost a century
(Mavin & Murray, 2010). This is in contrast to the assessment of NTS that has received
explicit attention relatively recently, and is seen as challenging (CASA, 2009e; Salas, Wilson,
Burke, Wightman, & Howse, 2006). Initially, NTS were included in training, which consisted
of intensive classroom based instruction, with courses running for up to two days, followed by
recurrent training occurring over half a day annually (O'Connor et al., 2008). Some airlines
introduced line orientated flight training (LOFT) in simulation training to counter some of the
failures in coordination and team communication identified in earlier aviation accidents
(Helmreich et al., 1999). Line orientated flight training was run in a similar fashion to a normal
flight, rather than using the numerous and different scenarios that were typical in previous
technical skills assessment. The normal flight would include flight planning and departure
procedures. The check captain, or simulator instructor, would then incorporate malfunction or
problems into the training scenario, requiring the pilots to deal with them in real time.
However, during the initial phase of NTS training, assessment was not conducted.

As outlined in Chapter One, the latter part of the 1990s saw the European Joint Aviation
Authority require training in NTS and - importantly - assessment of these skills. Investigations
to develop an assessment process found that models at that time, could not be used due to their
complexity or because the assessment process focused on crew, rather than individuals (Flin &
Martin, 2001; Flin et al., 2003b). The final system devised to assess NTS was called
NOTECHS (Flin et al., 2003b; JAA, 2002). NOTECHS was developed around the idea that
NTS were the cognitive and social skills of the flight crew members in the cockpit, not
directly related to aircraft control, system management, and standard operating procedures
(Flin et al., 2003b, p. 98). NOTECHS designers emphasised that, given the various cultures
operating in Europe, and the ensuing operating philosophies for aircraft, the metrics for NTS
40

Chapter Two
assessment should allow for these differences (Flin et al., 2003b). The final classification for
NOTECHS developed four categories: cooperation; leadership and managerial skills; SA; and
decision-making (see Table 6). Within NOTECHS, each category could be defined as either
social skills (cooperation and leadership and management) or cognitive (SA or decisionmaking). Each category was then further divided into elements to assist the assessment process.
Though NOTECHS may have developed particular elements that underpin each category, it is
argued that not all the elements reflect actual practice. For instance, the category of decisionmaking reflects heavily on the classical decision making model outlined in Section 2.3.2.
However, there is no reference to naturalistic decision making, the decision model most likely
to occur with expert performance.

Table 6 Categories and elements of NOTECHS

Taken from Flin et al (2003b, p. 102)

Category
Cooperation

Leadership and
Managerial skills
Situational
Awareness
Decision-making

Elements
Team-building
Considering others
Supporting others
Conflict solving
Use of authority and assertiveness
Providing and maintaining standards
Planning and coordination
Workload management
Awareness of aircraft systems
Awareness of external environment
Awareness of time
Problem definition and diagnosis
Option generation
Risk assessment and option selection
Outcome review

Another design parameter of NOTECHS was an emphasis on the observable behaviours of

individuals, rather than team or airline-specific assessment. The reasoning for this was that a
basic tenet of Crew Resource Management is that pilots should communicate in a manner that
41

Chapter Two
reveals their mental models and thinking processes to other crew (Flin et al., 2003b, p. 100).
Specific tools were developed to assist in achieving the assessment of observable behaviours.

In NOTECHS assessment, a qualitative criteria sheet was developed to assist in assessing NTS
(see Appendix 2). These verbal criteria described pilots good and poor performance of NTS,
requiring the check captain to match NTS performance with verbal descriptors. This approach
is based on a criterion-based form of assessment, which measures an individuals final
performance against specific criterion (Linn & Gronlund, 1995). This type of assessment has
several benefits. First, all individuals are assessed on the merit of their performance,
independent of how others perform. An individuals awareness of the explicit criteria under
which they will be assessed enables them to focus on their work, individual development, and
on intellectual growth (Sadler, 2005). However, Sadler (2005) outlines that no one-grade
description perfectly matches a specific skill; rather, assessors are required to make a best-fit
judgment. As will be discussed in the next section, the verbal descriptors used by NOTECHS,
while assisting in the articulation of specific skills, do not represent the total performance of a
pilot.

In addition to the NOTECHS system, other methods were developed to assist in assessing
NTS. In the United States, a progressive training and assessment program was developed to
allow training captains and check captains to implement training and assessment of both
technical skills and NTS. This program was called the advanced qualifications program (AQP).
The program was associated with four main goals. The first was to determine a pilots
readiness to fly in commercial operations. The second was the development of metrics to
appraise the effectiveness of the advanced qualifications program in pilot training. The third
goal was to reveal any individual pilots lack of performance. The fourth was to gauge a wider
range of deficiencies that may exist across an airline (Baker & Dismukes, 2002). Yet, it has
42

Chapter Two
been identified under this program and other similar programs, that the measuring of pilots
performance was a difficult task (Baker & Dismukes, 2002; Holt, Hansberger, & BoehmDavis, 2002; O'Connor et al., 2002). Additionally, the implementation of a program that
focuses on bringing technical skills and NTS together, but does not focus on the assessment
protocols, can create inconsistency. As Baker & Dismukes (2002, p. 206) outline, if there is no
agreement on standards, the performance ratings are a function of the particular instructor
conducting the assessment. Although the advanced qualifications program was developed to
assist in both technical skills and NTS assessment, the program appears to require more focus
in developing better measures to assess pilots performance.

2.6

Reviewing aviation assessment

Within Australia, airline operations are referred to as high capacity transport operations under
civil aviation order (CAO) 82.5 (CASA, 2009b). In accordance with this order, there are
detailed training and assessing requirements for pilots within an airline. Airlines are obliged to
have a certain number of flight crew, and to meet assessment standards (CASA, 2009b). The
most important section pertaining to the current study is contained within CAO 82.5, and
relates to provision for a training and checking manual. In Appendix 2 of this order, section 4
refers to the fact that airlines will have a training and checking manual for the use and
guidance of persons appointed within the training and checking organisation (CASA, 2009b,
p. 14). Most importantly, it states:
The training and checking manual must include the following material
(b) course outline, detailed syllabus, completion standards and specimen record
forms for each flight or simulator training program. (p. 14)

43

Chapter Two
The current study is about completion standards contained within the above order. It is
argued that although completion standards are well defined for technical skills, the NTS, which
have been identified as so important, are still underdeveloped. It must be assumed that given
that NTS do not have well defined metrics, the assessment of NTS relies on other measures,
such as intuitive judgments by check captains. This study therefore focuses on developing an
improved understanding of completion standards by investigating the criteria used to assess
pilots performance in line with the above order.

There appears a clear intent to implement improved methods for assessing the performance of
pilots NTS. As outlined in Chapter One, CASA initially instigated draft rulings that required
aircraft operators to provide training in crew resource management (CASA, 2002). More
recently, a notice of proposed rule-making and notice of final rule-making under Civil
Aviation Order 82.5 have been made, with airlines able to use a variety of methods to comply
with these rules, (CASA, 2008d, 2009d). However, recently these orders have been delayed.
An explanation for the delay by the current chief executive officer of CASA was that the
introduction of NTS assessment was seen as overly prescriptive (CASA, 2009e, p. 8).

The current methods used in airline pilot assessment programs do not completely satisfy the
requirement under CAO 82.5. Technical skills assessments, although detailed, are only one
area where assessment is required. Although there are measures such as NOTECHS that assess
NTS alone, and the advanced qualifications program, which attempts to address technical skills
and NTS assessment, it appears neither is able to achieve a suitable assessment approach
(Wilson et al., 2010). Researchers continue to question whether training, assessment, and
debriefing of pilots overlook NTS (Thomas, 2003). Further research into identifying the
criteria currently used by check captains to assess pilots performance, will assist in developing
a clearer set of assessment metrics.
44

Chapter Two
The aviation industry has continued to move towards continual improvements in safety. The
identification of NTS as a root cause in many accidents has created a need for enhancement in
assessment processes. It is clear that the industry is making attempts to address this
problematic situation, as seen by the push to implement NTS training and assessment to
counter the unfortunate effects of NTS deficits. However, the assessment of NTS is still seen to
be challenging. No model or framework has yet been devised that is able to be effectively used
in practice. One of the challenges clearly faced by the industry is the persistent view of
technical skills and NTS as being separate. Additionally, research that continues to fragment
NTS has proved less effective than anticipated. An approach that attempts to bring together all
skills into a single model may be a more successful way of moving forward in the current
debate. The field of education is one particular discipline that has achieved success in
integrating separate concepts into single models of teaching and assessment. This study will
examine educational theory and consider the ways in which the approaches used in the
discipline may assist in developing a single model for assessing airline pilots performance.

2.7

Focus of the current study

An improved understanding of the current criteria used by check captains to assess airline
pilots performance is important for airlines, and is the focus of the current study. Specifically,
the study is about developing an understanding of the criteria used to assess pilots undergoing
promotion to airline captain. To achieve this, it is important to define specific terms used in
assessment. Sadler (2005) argues that the terms criterion and standard are not
interchangeable. For example, an airline may be interested in determining a minimum standard
of performance required by those pilots that are being promoted to airline captain. To achieve
this, the airline may begin by investigating the individual criterion used during such an
assessment. For example, a pilot reaching a minimum standard may be required to demonstrate
45

Chapter Two
excellent decision making and flying skills, whilst communication and management skills
could be still developing. To better appreciate how such an investigation may be carried out
requires an understanding of criterion-based assessment.

In criterion-based or criterion-referenced assessment, an individuals performance is measured

against performance criteria and standards, rather than other individuals performance (Calkins
and Greg, 2008; Linn & Gronlund, 1995). The use of criterion-based approaches to assessment
focuses on individuals performance, rather than group norms. Utilising criterion-based
assessment, it is possible that no individual, or all individuals assessed, will reach a particular
performance standard. As mentioned in Section 2.5.2, in assessment of pilots non-technical
skills, qualitative criteria sheets are used to allow a check captain to make judgments about
performance against pre-specified criteria - a form of criterion-based assessment. It has been
suggested that using criterion-based assessment is an effective education instrument for
initiating institutional change and individual outcomes (Sean and Tierney, 2008). However,
simply utilising specific and detailed criteria does not always mean enhanced assessment
practices.

Sadler (2005, p. 189) points out that the use of criterion-based assessment depends upon
judgments made by professionals, and argues this practice should be understood and managed
rather than deplored. Sadler (2009) outlines that the difficulties encountered in judging
performance during assessment may lead to reduced levels of grade integrity. He suggests that
grade integrity can be maintained during assessment by adhering to three important factors.
First, evidence gathered must be of a valid nature. Second, assessment tasks should be rigorous
enough in depth and breadth so that resilient judgments can be made. Third, final grade coding
should be based on sound theoretical understandings, enabling final performance to be
accurately reflected in the grade achieved. For example, an individual pilots performance may
46

Chapter Two
be judged by a check captain as a pass, good standard, or an excellent standard. As outlined,
criterion-based measures for pilots have been developed for both technical skills and NTS (see
Appendix 1 & 2). However, it is suggested that assessing pilots performance with current NTS
criterion-based assessment measures may not be reflective of actual practice, or may have
limitations in the way in which they inform final grades. Sadlers (2009) final point, that final
grades need to be based on some level of theoretical understanding, is crucial to the current
study.

Sadler (2005) reviews four approaches used to develop standards: numerical cut-offs utilising
appropriate measurement scales, tacit knowledge, verbal description, and exemplars.
Furthermore, he suggests that the best method for describing standards is a combination of all
four approaches. Designing such a combination would begin by developing qualitative criteria
based on the recommendations of experienced assessors. Fortuitously, these criteria, for both
technical skills and NTS, are already developed (see Appendix 1 & 2). Subsequently, the next
step to constitute a standard is to determine what criteria to select, and their relationship with
each other (how the criteria are to be combined). As outlined by Sadler (2005, p. 193), this
would require the researcher to investigate the tacit knowledge that accounts for these
judgements. It is argued that the process of defining a standard, and in the case of the current
study, a minimum standard, must be commenced by investigating tacit judgments made by
check captains about pilots undergoing promotion to airline captain.

2.8

Summary

A major realisation in the aviation industry over the last few years has been in regard to the
high percentage of accidents being attributed to human factors. NTS training was developed in
an attempt to counter these statistics. To date, technical skills assessment has been robustly
47

Chapter Two
developed, although assessment deficiencies in NTS are seen as problematic. Assessment
metrics which combine both technical skills (refer Appendix 1), and NTS already exist; one
such metric is NOTECHS, which has criteria for co-operation, leadership and managerial
skills, SA and decision-making (refer Appendix 2). However, there is still difficulty in defining
and developing a holistic measure for pilot performance in general. More research on required
standards of performance is clearly required. The current study attempts to identify and
articulate criteria used by experienced check captains to assess airline pilots undergoing
promotion to airline captain. A research approach that enables an in-depth understanding of
these judgments becomes the next challenge.

48

Chapter Three

Chapter Three
Phenomenology as a research approach
3.1

Introduction

The previous chapter argued for research into how current airline pilots are assessed and, in
particular, the articulation of tacit judgments made by experienced check captains during a
candidates promotion to airline captain. In selecting a suitable research approach, Crotty
(1998) recommends that a social science researcher must have two thoughts at the forefront of
their mind. First, what methodology is appropriate for a particular research problem? In the
case of the current study, how does one go about researching the assessment of pilots
performance during promotion to airline captain? Second, is the methodology being used in a
way that is consistent with its underpinning theoretical basis? This chapter begins by tackling
the second question first. It commences by exploring the theoretical assumptions of research
approaches such as positivism, social constructionism, and subjectivism. This exploration
enables a case to be made for using phenomenology, and underpinning theory of social
constructionism, as an appropriate approach to studying pilots performance. To better
understand phenomenology, the chapter will describe phenomenology as a philosophy,
including its theoretical basis and historical roots. The descriptive phenomenological
psychological method used in the present study is then explained with the required steps in this
research approach. Finally, validity and reliability are discussed when using phenomenology as
a methodology.

49

Chapter Three

3.2

Researching the assessment of pilots performance

Study of knowledge has been at the forefront of philosophy for millennia. In particular, much
thought has been given to defining the concept of knowledge, with prominent thinkers
arguing that knowledge is a factual truth, free from the subjective views of human
consciousness. For example, during the eighteenth and nineteenth century, or age of reason
(Crotty, 1998, p. 18), philosophers such as Compte saw society advancing to a point where all
knowledge would be scientific, through a positivistic epistemology (Kenny, 2007). Science
was conceptualised as a system of observation and experimentation, the outcomes of which
would be used to form an organised body of information, otherwise defined as knowledge
(Crotty, 1998). Epistemology is a branch of philosophy that has been focused on the nature,
sources, and limits of knowledge; it has been defined as the theory of knowledge, especially
with regard to its methods and validation (Moore, 2004, p. 422). Positivism became prominent
during the Enlightenment period of the 17th century, stating an assurance of unambiguous and
accurate knowledge of the world (Crotty, 1998, p. 18). Positivism is well embedded in the
natural sciences, like biology and physics, and sees knowledge as separate to human
consciousness with proponents asserting that the only true way to obtain knowledge is through
sense experience and positive verification (Crotty, 1998). For example, in the science of
aerodynamics, data may be viewed as knowledge in the way that positivists would define it, as
data is based on information obtained via the scientific method, independent of subjective
interpretation. Some of the success of psychology and medicine over the last century has been
fuelled by views that significant claims can only be substantiated with the type of empirical
data that these disciplines strive to acquire. Some of the advancements of knowledge made
during the last centuries, especially within natural science research, were, largely a result of
studies underpinned by a positivist view (Schn, 1983).

50

Chapter Three
Although positivism has proven beneficial to the advancement of natural science investigations
over many years (Crotty, 1998), a continuum of epistemological theories exists, ranging from
positivism at one end, to subjectivism at the other, as Figure 11 shows. Positivism sees subject
and object as separate (Moore, 2004). Subjectivism, on the other hand, holds that meaning is
imposed upon objects by individuals or subjects (Crotty, 1998). Subjectivism rejects the view
that object and subject are separate, so that there is no objective truth. Rather, humans impose
all meaning through their experiences.

Positivism

Social
Constructionism

Subjectivism

Figure 11 Epistemological theories of knowledge

Modified from Crotty (1998, p.5)

Sitting between positivism and subjectivism on this continuum is constructionism.

Constructionism does not view objects and subject as separate, nor does it advocate subjects
impose all meaning; rather, it sees object and subject having a closer relationship, with
individuals constructing meaning as they experience the world in which they live (Crotty,
1998; Silverman, 2001). Constructionisms positioning is determined by the degree to which
the approach focuses on either objectivist or subjectivist paradigms (Crotty, 1998).
Constructionism has many theories under its banner, such as interpretivism, phenomenology,
and hermeneutics (Crotty, 1998). Recent decades have seen an increasing emphasis on the
social nature of knowledge within constructionism. While individuals construct meaning of the
world, they do not do this in total isolation of others around them, or of those who have gone
before them. Taking into account social and historical principles implies a social context to the
51

Chapter Three
construction of knowledge - in other words, social constructionism (Crotty, 1998; Sandberg,
2001). So, the way in which knowledge is defined may differ depending on the perspective
taken by proponents of positivist, social constructionist, or subjectivist theories.

During investigation into the most appropriate approaches to conducting research on the topic
of pilot performance assessments, there was a distinct tension found in the wider literature.
This tension relates to the differing philosophical approaches as to the way in which
knowledge is viewed. For example, during advances in positivistic research, a growing group
of philosophers and professionals began questioning research based on positivism. Positivism
was seen as denigrating earlier insights, with various phenomena not being able to be described
in positivistic terms, such as the concepts of instability, uncertainty, value, or conflict. The
dilemma for positivists was that they were compelled to exclude phenomena they have learned
to be central to their practice (Schn, 1983, p. 42). The approach of neglecting the
consciousness of humans was being seen as problematic (Crotty, 1998). Philosophers such as
Husserl were critical of the positivistic view. Husserl posited that a subjects consciousness
could not be separate from the world, but was the means by which we experience the world
(Moran, 2000). For Husserl, nothing could be even spoken about or witnessed if it did not
come through someones consciousness (Giorgi & Giorgi, 2008, p. 26). For Husserl, the
subject is always conscious of something, be it real or imaginary (Moran, 2000).

In more recent times, a growing number of researchers have moved away from positivism, in
areas like education (Latham, 2001), professional development (Brearley, 2001; DallAlba,
2002; Sandberg, 1994, 2000; Webster-Wright, 2006), and health and wellbeing (Lempereur &
Lauri, 2006; Welch, 2001). These theorists rejected attempts to deal with object and subject as
separate.

52

Chapter Three
Whilst investigating suitable approaches to conducting the current study, it became evident that
there are certainly limitations to studies conducted solely within a positivistic framework.
Indeed, it became apparent that certain difficulties would present themselves in trying to
conduct research into check captains complex judgments, based on a research philosophy that
saw this complexity as being void of their experience, attitudes and beliefs. On the other hand,
social constructionism provided the balanced framework being sought for this study. In
totality, this approach was selected as the one that provided the appropriate means of
grounding the epistemological basis on which the study design was founded. Thus, a research
methodology underpinned by social constructionism was selected.

Within this framework, phenomenology was seen as an appropriate way to investigate pilots
performance. Though phenomenology is not restricted to human subjects, it is generally seen
as an approach that attempts to illuminate descriptions of phenomena, or in other words, of
things as they are perceived by individuals (Barnacle, 2001; Moran, 2000). Husserl, the
architect of phenomenology, saw it as the study of the ways in which things appear to
consciousness (Brearley, 2001, p. 77). Phenomenology was originally a philosophy, although
it is now also an approach to conducting empirical research (Moran, 2000). In empirical
studies, phenomenology gathers deep descriptions via qualitative methods such as interviews
or case studies (Barnacle, 2001; Giorgi & Giorgi, 2008).

There are three central reasons supporting the use of a phenomenological approach in this
study. First, it enables researchers to maintain a holistic approach to research. As argued in
Chapter One, current approaches to assessment of pilots performance separate skills into
smaller areas for assessment. However, such separation is problematic. In contrast,
phenomenology sees knowledge as interrelational and structural, interwoven in networks
(Kvale, 1999, p. 101). Phenomenology sees knowledge as related, interlinked, socially
53

Chapter Three
constructed and personal. In investigating phenomena, Giorgi and Giorgi (2003) assert that
phenomenology:
aims to clarify situations lived through by persons in everyday life. Rather than
attempting to reduce a phenomenon to a convenient number of identifiable variables,
and control the context in which the phenomenon will be studied, phenomenology
aims to remain as faithful as possible to the phenomenon and to the context in which it
appears in the world. (2008, pp. 27-28)
Phenomenological research allows a researcher to investigate phenomena - or in the case of the
current study, it lends itself to study the criteria used by check captains to assess pilots
performance, such as SA, decision making, management, or any other criteria that may yet be
discovered. When assessing pilots, check captains need to be familiar with assessment criteria
in depth, as they are based on rich descriptions of their own experiences.

Second, being holistic in its approach, phenomenology captures and explores in-depth the rich
descriptions of lived experiences. For example, whilst identifying individual criteria which are
beneficial, there is a vast depth of knowledge and experience that check captains develop over
many years of assessing, which give greater insight into the criteria, and how they are used.
Furthermore, the complex judgments made by check captains about pilots performance rely to
some extent on intuition. Through phenomenological research, it is possible to investigate
these complex judgments, and explore first-hand experiences by individuals, and describe
events which require discernment and judgments (Giorgi & Giorgi, 2008, p. 28). It is through
these descriptions that depth of experience can be articulated.

Finally, for the purposes of this study, the phenomenological approach provides a framework
which includes human factors that positivists have excluded as being irrelevant or distracting.
These arguments against positivistic theory are supported by other researchers who have
54

Chapter Three
investigated performance within work settings, using phenomenology as a research
methodology (Sandberg, 2000, 2005; Webster-Wright, 2006).

3.3

Phenomenology as a philosophy

In this section, phenomenology as a philosophy is reviewed. Husserl is recognised as the

founder of the phenomenological movement with his first edition of Logical Investigations
(Husserl, 1970). Although his initial philosophy was influenced by Brentano, other works, such
as those by Kant, Hegel, and Mach, were also influential in Husserls writing (Kenny, 2007).
Husserl viewed the subjects consciousness not as separate from the real world, but as the
means by which we experience the world. As outlined above, Husserl viewed subjects as
always being conscious of real or imaginary objects (Moran, 2000). This directedness of
consciousness was referred to as intentionality: When the mind becomes conscious of
something, when it knows something, it reaches out to, and into, that object (Crotty, 1998, p.
44). This sense of intentionality has a different meaning as the everyday English term (i.e.,
intentional, deliberate or planned). Rather, it has its roots in Latin tendere, which means to
tend or to direct oneself to (Crotty, 1998, p. 44). Crotty writes that intentionality posits a
quite intimate and very active relationship between the conscious subject and the object of the
subjects consciousness (1998, p. 44).

For Husserl, intentionality required two things. First, consciousness must be about something,
such as an airplane, love, or make believe; you cannot have consciousness about nothing.
Second, consciousness must belong to someone, such as me, the researcher, or you, the
reader. Husserl outlined that consciousness could be described as an act or, more importantly,
my act or your act (Kenny, 2007). Accordingly, subject and object are not separated, but

55

Chapter Three
interrelated. As Giorgi (1997) describes it, the meaning of a subject implies a relationship
with an object.

Husserl posed a central philosophical question: how is objectivity established within

consciousness (Moran, 2000)? A complexity that Husserl had to tackle within his
phenomenological philosophy was that consciousness about the world must therefore be part of
the world. Husserl confronted this by developing his theory on life-world (Kenny, 2007). Lifeworld was the world of pre-theoretical experience which is also that which allows us to
interact with nature and to develop our own cultural forms (Moran, 2000, p. 181). From the
life-world, Husserl was then able to move away from the subject-object dilemma, and focus on
a particular phenomenon. A phenomenon was the relationship between object and subject,
rather than how subjects experience an independent object.
One must not think of objects as existing exactly in the manner in which they are given
in the view from nowhere. All objects are encountered perceptively; all conscious
experience occurs in a temporal flow, the nature of which must be recalled in any
analysis of human perception. (Moran, 2000, p. 13)
For example, during the current study, pilots performance would be considered the object.
There is the check captain, or subject, who has years of experience with assessing pilots
performance. The relationship between the check captain, and the pilots performance is
called the phenomenon. Additionally, as Moran (2000) states, objects dont just exist
independently of the subject, so that the subject/object relationship is seen in one way. In the
example above, check captains may differ in their assessment of pilots performance. So for
Husserls life-world, the subject-object impasse was able to be resolved by focusing on
phenomenon.

56

Chapter Three
Husserl asserted that phenomenology could be regarded as a philosophy. Part of the appeal of
phenomenology is the rigorous defense of the fundamental and inextricable role of
subjectivity and consciousness in all knowledge and descriptions of the world (Moran, 2000,
p. 15). As such, Husserls phenomenological philosophy established methods of describing the
role which consciousness played in the development of knowledge (Moran, 2000). Husserl
outlined three steps to describe the role of consciousness: phenomenological reduction,
description, and structure of phenomenon.

3.3.1 Phenomenological reduction

In Husserls terms, reduction is a way of making research more precise (Giorgi, 1997).
Reduction begins with bracketing previous perceptions, whilst setting aside unquestioned
assumptions about the world. It requires a philosopher to reserve their biases, and set aside
their presuppositions in order to allow meaning to come from the data, rather than from takenfor-granted interpretations (Crotty, 1998; Giorgi & Giorgi, 2003; Hein & Austin, 2001; Moran,
2000). This bracketing is a rigorous process that occurs throughout the research.

3.3.2 Description
Once a philosopher has entered into the phenomenological reduction, the aim is to then
describe phenomena in the life-world. As Giorgi (1997) explains, linguistic descriptions of
original phenomena in the life-world must show the truest possible level of fidelity. Giorgi
(1997), citing Merleau-Ponty (1945/1962), suggests that this fidelity is achieved through a
focus on description rather than explanation. That is, the philosopher phenomenologist must
move away from the interpretations and attempt to describe, rather than explain, phenomena in
the life-world.

57

Chapter Three
3.3.3 The structure of a phenomenon
The structure in phenomenological investigation involves the philosopher finding ways to
describe the important and inherent aspects of a phenomenon. Initially, the phenomenon
described may at the outset appear to have one meaning. However, to understand the structure,
the philosopher must begin a process of free imaginative variation (Giorgi & Giorgi, 2003, p.
248). As Giorgi (1997) suggests, this variation is about continually changing how a
phenomenon is viewed, thus enabling new and undiscovered meaning to evolve. This continual
refinement assists in finding a structure that attempts to best describe the phenomenon.
However, we never reach a final structure, as it is dependent on our understandings, which
continuously develops.

3.4

Phenomenology as an approach to empirical research

Phenomenology is regarded as a valid way to investigate events experienced by individuals in

their daily lives (Giorgi, 1994, 1997; Giorgi & Giorgi, 2003). The steps outlined in the
previous section describe the process of conducting philosophical phenomenology. On the
other hand, empirical research may use a phenomenological methodology, but it requires a
different approach (Giorgi & Giorgi, 2003). Giorgi and Giorgi (2003) outline the differences
between the two approaches, stating that empirical phenomenological research is conducted
with participants, while philosophical phenomenology is typically carried out by a researcher
in isolation. As would be expected, empirical phenomenological research draws a great deal on
philosophical phenomenology, though it needs to be adapted to allow it to be used in empirical
research.

Empirical phenomenological research relies on three main sources of data: the researchers
reflection on conducting the research, participants descriptions, and descriptions of
58

Chapter Three
phenomena from sources such as written literature, text, and television (Hein & Austin, 2001)
and, sometimes, from observations (Dall'Alba, 2009; Sandberg, 2005; Webster-Wright, 2006).
The current study was based on descriptions obtained from participants (who were check
captains). The research methodology using descriptive phenomenological psychological
method is based on Giorgi and Giorgi (2003, p. 243). The descriptive phenomenological
psychological method describes seven steps:
1. Collecting verbal data via semistructured interview
2. Transcribing verbal data
3. Reduction
4. Reading of interview transcripts
5. Dividing the data into parts
6. Ordering data into disciplinary language
7. Expressing the structure of the phenomenon
These seven steps will now be outlined.

3.4.1 Collecting verbal data via semistructured interviews

Phenomenology attempts to describe a phenomenon, or the relationship between subject and
object; in other words, it describes the way in which something is experienced by someone.
This necessitates those involved in empirical phenomenology to obtain participants
experiences of the life-world, to be described as wholly and faithfully as possible. As the
researcher is often not the one who experienced the phenomenon, the empirical researcher
obtains descriptions from others (participants), in the form of second-order descriptions
(Giorgi & Giorgi, 2003, p. 251). In obtaining such descriptions, the interview becomes the
research instrument that is a highly useful tool for data collection. For this reason, the interview
became the preferred approach for the present empirical phenomenological study.

59

Chapter Three
The interview is a flexible way of collecting descriptions from participants. Kvale (2006, p.
483) outlines the conventional view that an interview is a joint endeavour where egalitarian
partners, through conversation, search for true understanding and knowledge. In relation to
phenomenological interviews, Smith and Osborn (2008) describe good interviews as those in
which movement from general to specific questions and descriptions occur seamlessly.
Although an interview may be used in phenomenological research, social constructionism - the
theory underpinning phenomenology - rejects the notion that language is a true representation
of the life-word (Sandberg, 2001). Accordingly, a researcher needs to carefully consider the
implication of using an interview as a data collection instrument. For example, questions need
to act as a catalyst for participants descriptions of their experiences within a particular area of
interest to a researcher. Open-ended questions are appropriate for this need (Smith & Osborn,
2008). Even so, just because open-ended questions are used, this does not automatically imply
that participants give meaningful, unbiased information in an interview (Freebody, 2003;
Silverman, 2001). For example, Foddy (1995) reported that an interviewee later regretted not
providing more detail or better accounts in response to particular questions. As a consequence,
open-ended questions do have some disadvantages that need to be taken into account.

To counter the limitations of open-ended questions, semistructured interviews could be used.

These interviews are planned around pre-arranged open-ended questions that can increase the
control a researcher may wish to have in the interviewing process. This can be done firstly by
allowing participants, via open-ended questions, to adequately express their descriptions in
full. Secondly, the semi-structured interview style allows researchers to ask follow up
questions, or follow a line of discussion that was not anticipated (Smith & Osborn, 2008).

Once the style of interview is determined, the interview format and question development must
be considered. The interview format or interview schedule includes types of questions and a
60

Chapter Three
suggested order in which they will be asked. The interview schedule has two benefits. First, it
compels the researchers to consider intended outcomes of the interview. Second, the researcher
is able to refocus an interview if it moves in unanticipated ways, as well as to ask follow up
questions for clarification (Smith & Osborn, 2008). Thus, the semistructured interview has
some advantages for deep and rich collection of data.

Although researchers may attempt to take into account as many challenges as are likely to
arise, unexpected problems may present themselves during the interview process. Two
problems in particular are associated with phenomenological research. Problems may be
encountered when participants make errors recalling original phenomena. For instance, a
participant may discuss a past situation, which after later reflection, was described
inaccurately. This dilemma can be exacerbated with scientific phenomenological reduction (see
below), as researchers will be open to all types of descriptions (Giorgi & Giorgi, 2008).
Another problem is that, for a variety of reasons, a participant may be reticent to fully divulge
details, or may deliberately distort the realities that they experienced. It is even possible that
the participant may be deceitful, and this may be recognised as such in the transcripts and
structures, or by using techniques during the interview, for example, follow up questions. This
issue will also be taken up further in the chapter when discussing validity in phenomenological
research. However, Giorgi and Giorgi (2008) suggest that there is no foolproof method for
countering deceit in this or many other research methodologies.

A final issue that must be considered is in regard to the level of familiarity a researcher may
have with participants. In qualitative studies, this is often referred to as insider or outsider
status. An insider has membership of a particular group - such as race, gender, class or other
similarities - that enables the researcher to have personal knowledge of the group being
researched, whereas an outsider does not have this intimate knowledge (Mercer, 2007). It
61

Chapter Three
appears the literature is undecided on the benefits and limitations of insider/outsider research,
as Mercer (2007, p. 5) calls it, the double-edged sword. For instance, it is argued that only an
outsider can obtain purely objective results during research (Mercer, 2007 citing Simmel,
1950), whilst in contrast, it is suggested that particular activities will always remain hidden
from the outsider (Flick, 2006). Similarly, it is argued that being an insider may cause a
reduced level of information to be divulged for fear of being judged (Merver, 2007). However,
Robson (1993, p. 298) outlines it is beneficial to an investigation if the researcher has some
level of street credibility.

An interesting dilemma for those trying to find an objective appraisal for dealing with insider
and outsider research are those arguments that consider both doctrines to be fallacies (Mercer,
2007, p. 5). For example, Merton (1972), cited by Mercer (2007), stated that we cannot simply
categorise human beings as belonging to, or not belonging to, one group. From this, Merton
contends that we cannot assume that membership of a particular group produces similar
insights or understandings. On the other hand, it is argued that just having membership enables
a researcher to obtain richer data during a study (Mercer, 2007).

One way a researcher may allow for the insider/outsider influence is to review the researchers
position. For instance, the effect a researcher has on a study has been widely argued within
areas of research, such as feminism. As such, the term positionality has been coined to reflect
on the consequences a particular researcher has on the production of knowledge (Burns &
Walker, 2005). It has been conceded that knowledge produced from an acknowledged
standpoint is less distorted, more visible and hence revisable than knowledge which erases its
partiality (Burns & Walker, 2005, p.67). This standpoint allows researchers to openly
acknowledge their position in relation to a particular study. Though there may be advantages to
not being a member of the group being researched during a study, there appears a real
62

Chapter Three
argument that familiarity, or an insiders perspective, may unveil more hidden knowledge
during a study. However, the researcher must acknowledge this membership, so that there is
greater visibility on how the knowledge was produced.

3.4.2 Transcribing verbal data

After interviews have been conducted, the next stage requires verbatim transcriptions of the
interviews. This is done by listening to the interviews and transcribing all spoken words
verbatim into written form. This is the point in the research where data collection is complete.
The data is then available to the researcher for reading (Giorgi & Giorgi, 2003).

3.4.3 Reduction
Problems arise when a researchers interpretations differ to that of participants. Even Husserl
was acutely aware of the difficulty obtaining descriptions, and he was uneasy about researchers
assumed meanings based on their own experiences (McKenna, 1982). In order to reduce bias
within a study, a researcher undergoes a process called reduction. Husserl described this as
attitudinal modification, removing oneself from conventional thinking (Giorgi & Giorgi, 2003,
2008; Moran, 2000).
When we encounter familiar objects we tend to see them through familiar eyes and
thus often miss seeing the novel features of familiar situations. Hence, by
understanding that the given has to be seen merely as a presentational something
rather than the familiar object that always is there, new dimensions of the total
experience are likely to appear. (Giorgi & Giorgi, 2003, p. 249)

Giorgi & Giorgi (2008) assert that the philosophical phenomenology method enables a
researcher to enter into reduction from the start of the research, though they suggest this is less
practical in empirical research, where data is yet to be collected. As a consequence of the
63

Chapter Three
researcher not entering immediately into reduction, particular interview techniques are used to
draw out in-depth description (see later section on validity 3.5.1 and reliability 3.5.2). Once
interviews are complete and transcribed, the empirical researcher then enters into
phenomenological reduction by bracketing as many assumptions and as much bias as possible
(Giorgi & Giorgi, 2003; Hein & Austin, 2001). During analysis of data, the reduction limits the
likelihood of a researcher failing to critically evaluate some descriptions given by participants.
Even though seemingly obvious descriptions are made, to make research more valid,
researchers must ask whether descriptions could exist in other ways (Giorgi & Giorgi, 2003,
p. 249).

3.4.4 Reading of interview transcripts

Thorough and purposeful reading of transcripts enables the researcher to gain an overall sense
of the data. Giorgi and Giorgi (2008, p. 34) recommend that researchers get within the attitude
of the scientific phenomenological reduction, and be mindful of the phenomenon being
studied. At this phase in the analysis, a researcher is not required or encouraged to thematise
the data. Rather, this juncture facilitates the researchers immersion into the life-world
descriptions given by participants (Giorgi, 1997). It is during the next phase of the process that
data begins to be transformed into more usable information.

3.4.5 Dividing the data into parts

This stage requires participants descriptions to be divided into meaningful parts, or meaning
units. Meaning units may be single words, sentences, or paragraphs, in the original state
spoken by participants. As Giorgi (1997, p. 245) describes, since phenomenology is interested
in meanings, the basis of division into parts is meaning discrimination. Whilst reading
transcripts, the researcher may notice a change of meaning. If a so called transition is
experienced, the researcher may place a mark after text containing the meaning, thus creating
64

Chapter Three
meaning units. In line with the epistemology that grounds phenomenological research,
meaning units are not objective units, but originate from the researchers analysis of the text
(Giorgi & Giorgi, 2008).

3.4.6 Ordering data into disciplinary language

Once separate meaning units have been created, the researcher is able to transform each of
them into disciplinary language. For instance, empirical phenomenological research is
conducted with people describing their particular experiences. Descriptions are generated
during conversations about the research topic. For example, the researcher may ask individuals
to describe roles, functions, or job descriptions in their given profession. At some point,
however, these meaning units need to be transformed into a language that is reflective of the
researchers discipline (Giorgi, 1997). Herein the reduction comes to the forefront, with a
researcher bracketing presuppositions and interpreting meaning, rather than imposing opinion.
Once the interview is broken into meaning units, it is written into disciplinary language
through a process of transformation. Giorgi and Giorgi (2008, p. 245) advise that the purpose
of transformations is to render the implicit, explicit. Generally, more than one transformation
occurs, with similar units being combined and summarised into small, succinct descriptions.
Whilst writing transformations, it is recommended that the use of scientific jargon be avoided
(Giorgi & Giorgi, 2003, p. 253). The life-world is, as Giorgi (1997) suggests, pre-scientific and
pre-theoretical. Once life-world descriptions are converted into a meaningful format, they
facilitate further analysis.

3.4.7 Expressing the structure of the phenomenon

The final stage of analysis requires researchers to develop a deeper understanding of the data.
To do this, the researcher is required to develop a structure/s, using disciplinary language.
Written structures describe how individuals and groups make sense of their world. They are
65

Chapter Three
succinct ways of pulling together common experiences. Structures vary with regard to the
complexity of the study and the number of participants. Increasing participant numbers tends to
provide richer description, and consequently structures also tend to be richer (Giorgi & Giorgi,
2008). Even so, it is important to understand that as participant numbers grow, structures do
not necessarily grow proportionately. For example, from two or three participants, two or three
structures may crystallise, whereas with 20 participants, five structures may arise (Smith &
Osborn, 2008).

The number of structures within individual research projects is an important consideration.

Single structures are adequate; Giorgi (1997) recommends developing single structures for
simplicity. Nevertheless, he advises that structures should never be forced together, but
caressed. It is important at this stages that not only are individual constituents of structures
described, but also the relationship between the constituents (Giorgi, 1997). When two
structures present themselves with the same constituents, it is the differing interrelationships of
parts that differentiate the structures (Giorgi & Giorgi, 2008).

Once all structure/s have been developed, the researcher begins to determine if specific
constituents can be removed. If certain constituents can be removed from structures, and
structures remain, then the constituents can be considered as accidental rather then essential
(Giorgi & Giorgi, 2003, p. 246). It is when structures collapse after the removal of a
constituent that that constituent is essential. A house could be used as an analogy. If we began
stripping away parts of the house, when does it stop being a house? For instance, the patio
would not be essential for a house, nor a garage door. The roof would, however, be considered
essential. While the roof is essential, the patio is accidental.

66

Chapter Three
The method outlined above appears linear in its approach. The reality is that, in practice, it
resembles a dynamic process that requires the researcher to move backwards and forwards
between steps to better understand what has been presented and described by participants.
Some of the validity and reliability issues rely heavily on this iterative process.

3.5

Validity and reliability in phenomenological research

Research based on methodologies which adopt qualitative approaches have been criticised
when the techniques used to analyse data are ill defined, and hence lead to concerns of invalid
and unreliable results (Miles, 1979). For instance, during the early development of social
science research, there were heavy influences from natural science methodologies. This created
what Giorgi (1994, p. 191) saw as mixed discourse in investigations, and researchers found it
difficult to detach from traditional natural science paradigms. To assist in this dilemma, Giorgi
(1994, p. 191) argues there must be approaches that can help the development of the field of
qualitative research and enhance its legitimation. An important argument that Giorgi makes is
that even when overlaps occur between the methodological approaches of natural science and
social science, comparisons can only be done by respecting the theoretical differences
(1994, p. 194). Empirical phenomenology therefore uses methodological strategies to
strengthen its findings, rather than abandon concepts which are critical to its philosophy, but
are used nevertheless, in order to align with natural science methodologies.

3.5.1 Validity
To counter criticisms regarding validity that may be made against phenomenological research,
Sandberg (2005) has proposed that three areas strengthen the validity of the phenomenological
approach. These areas are named communicative, pragmatic, and transgressive validity. It is
argued that each area allows for perceived fulfillment, fulfillment in practice and
67

Chapter Three
indeterminate fulfillment respectively within interpretive research, with each being expanded
below (Sandberg, 2005, p. 54).

3.5.1.1 Communicative validity

Communicative validity describes how the researcher is able to attain perceived fulfillment
(Sandberg, 2005, p. 54). It is concerned with acquiring and validating knowledge, asserted by
the participant during dialogue (Kvale, 1995, 1999). To achieve communicative validity,
Sandberg (2005) argues three areas must be satisfied for truth claims to be made. First, valid
knowledge claims presuppose an understanding between researcher and research participant
(Apel, 1972, as cited in Sandberg 2005, p. 54). Thus, yielding valid data requires an
understanding between researcher and participant about research purpose, or what Sandberg
(2005, p. 54) calls a community of interpretation.

Second, during analysis of data, maintenance of communicative validity requires preserving

coherence (Sandberg, 2005). Kvale (1995) argues that the coherence criterion centres around
the logic and consistency of an argument or statement. The more consistently a theme or
description emerges, the higher its communicative validity. This does not mean other views are
not considered, as this would not be consistent with the phenomenological methodology.

Third, Sandberg (2005) suggests communicative validity can be achieved by discussing

findings with participants and other researchers. Referring to Gadamer (1960/1994), Sandberg
(2005, p. 56) states that truth is to a large extent achieved through dialogue between people.
Sandberg continues by outlining that the life-world reflects continual refinement and
challenging views of truth. As he explains, life-world is subjectively interpreted, but these
interpretations are also compared between subjects. However, caution should be used with an
over-reliance on inter-subjective validation as a means of validating interpretations. An over68

Chapter Three
reliance on others suggests an unwillingness by the researcher to be accountable for his/her
findings (Kvale, 1995).

Even though a researcher may have addressed the three areas in obtaining communicative
validity, this type of validity still relies on the interpretation of others understandings on the
basis of what they say (Giorgi & Giorgi, 2008; Smith & Osborn, 2008). Smith and Osborn
(2008, p. 53) call it a double interpretation:
the participants are trying to make sense of their world; the researcher is trying to
make sense of the participants trying to make sense of their world.

Indeed, Smith and Osborn suggest that interpretations of individual experiences rely on a
theoretical acceptance of a link between talk and lived experience of individuals. No matter
how complex or confusing they may seem, the aim is to be true to the descriptions that are
obtained.

3.5.1.2 Pragmatic validity

While communicative validity assists the researcher to obtain knowledge through dialogue
(Kvale, 2001), pragmatic validity confirms what people say they do and what they actually
do, and enables a researcher to obtain fulfillment in practice (Sandberg, 2005, p. 56).
Pragmatic validity endeavours to capture knowledge as it occurs in practice. Achieving
pragmatic validity assists in dealing with the weaknesses that can be found in communicative
validity (Kvale, 1995, 1999). As Kvale (1999) outlines, there is an array of meanings situated
in differing cultures and practices, which makes it difficult to interpret what people do.
Importantly, the underlying foundation of pragmatic validity is that actions speak louder than
words (Kvale, 2001, p. 21). Pragmatic validity can be accomplished by asking follow-up
questions in order to obtain concrete examples of participants experiences. Furthermore,
69

Chapter Three
Sandberg (2005) suggests the use of deliberate misinterpretation as a way of checking
pragmatic validity. For example, during interviews, another question can be asked which
incorrectly quotes the participants previous descriptions. The participant may then correct the
researcher, and reiterate initial statements, thereby improving pragmatic validity.

Another method that may be used to obtain pragmatic validity is to test actual interpretations
and findings in practice (Sandberg, 2005). In some cases, this would require a larger study, or a
separate study to be conducted.

3.5.1.3 Transgressive validity

Both communicative and transgressive validity assist the researcher to comply with perceived
fulfillment and fulfillment in practice (Sandberg, 2005, p. 54). However, Sandberg argues
that relying on these two areas only may reduce levels of validity.
This means that the criteria of communicative and pragmatic validity may encourage
the researcher to overlook various forms of ambiguity, complexity, and multiplicity in
the lived experience investigated (2005, p. 57).
A researcher can claim transgressive validity through indeterminate fulfillment; not by
searching for interpretations that appear regularly but rather via disagreement, tension, and
uncertainty that cannot be resolved within the discussion (Sandberg, 2005, p. 54).

A technique for maintaining transgressive validity is to take a strong contradictory view of

variations which occur in descriptions. Lather (1993), cited in Sandberg (2005), uses concepts
like irony and contradiction to test for transgressive validity. For example, original
interpretations are deliberately opposed, via original transcripts, and support is sought as a way
of upholding these new interpretations. If such support is uncovered, the original interpretation
could be rejected. At this stage, if support is not gained, original interpretations would hold up
70

Chapter Three
to greater scrutiny. Yet, phenomenologists must never lose sight of the fact that they do not
seek objective truth, but try to describe experiences of objects or events (Smith & Osborn,
2008).

Communicative validity enables a researcher to determine ways for a participant to articulate

their experiences. However, it does not confirm that what is described is in fact what actually
occurs. This is accomplished through pragmatic validity. However, together, communicative
and pragmatic validity do not allow for inconsistencies to be actively sought - this is
accomplished by transgressive validity (Sandberg, 2005). Sandberg (2005) concludes that the
weaknesses of each are reduced by the strengths of the other.

3.5.2 Reliability
Reliability, traditionally of interest within the natural sciences, focuses on the duplication of an
investigation (Babbie, 1994; Sandberg, 1997; Silverman, 2001). For example, in some research
circles, interjudge reliability is used to measure how effectively one researcher could
replicate anothers investigation (Sandberg, 1997, p. 205). However, in qualitative research,
differing views occur regarding how reliability should be judged (Rolfe, 2006; Sandberg,
2005). Some researchers reject the idea that reliability, stemming from natural science, is an
appropriate metric for social science research (Sandberg, 1997; Silverman, 2001; Yardley,
2008). Some of the rejections are made on a number of grounds, including the idea that
interjudge reliability does not take into account the researchers procedures for achieving
faithful descriptions of the individuals conception of reality (Sandberg, 1997, p. 206). As a
result, findings using qualitative approaches are made reliable through continual and explicit
explanation of reasons behind the researchers actions. Furthermore, phenomenological
research does not aim to be like natural science research; rather, it claims knowledge, obtained
using its particular methodology, is valid and reliable (Kvale, 1999). However, there may be
71

Chapter Three
misgivings, if researchers hide behind the above claims, and do not, at least, describe how
reliable results can be achieved and demonstrated.

Sandberg (2005) outlined five ways that interpretive studies can achieve and demonstrate
reliable research. First, the researcher must show how they devised interview questions.
Second, there needs to be an outline of how and why participants were selected. Third, the
researcher should detail procedures and processes of data collection. Next, the researcher must
detail how data is analysed. This inevitably becomes more comprehensive when more than one
researcher is involved in the analysis (Miles, 1979; Yardley, 2008). Finally, the findings will
provide sufficient detail and examples to enable a reader to understand how the results were
obtained.

Even if a researcher follows the above five steps, it is impractical to present, in some cases,
hundreds of thousands of words of data and analysis to a reader. As a consequence, qualitative
researchers provide only appropriate evidence to the reader so they can develop a clear picture
of the research. Even so, to provide a level of assurance, it is still necessary for researchers to
provide a paper trail, if interested peers wish to confirm or audit the research (Yardley, 2008,
p. 243). Although qualitative researchers may argue that natural science reliability measures
are not appropriate for their investigations, they must be able to demonstrate how their research
is reliable. An important point that must always be considered is that research which takes on
only those interpretations that agree with the researchers ideas is unethical (Sandberg, 2005).

3.6

Summary

This chapter has outlined the theoretical framework that underpins the present study.
Importantly, this study has moved away from the traditional approaches to aviation research,
72

Chapter Three
entrenched within positivism, and adopted the approach of social constructionism as a suitable
underpinning theory. To this end, this chapter has explored phenomenology as a philosophy,
and as the basis of a research methodology. This foundation has been built with the assistance
of Giorgis empirical approach to research, as well as by the methods of enhancing validity and
reliability in interpretive methodologies. The next chapter details how the phenomenological
approach is used as the basis of a research methodology to the investigate pilots performance
during promotion to captain.

73

Chapter Four

Chapter Four
Method
4.1

Introduction

The method used to conduct this research, including the procedures followed and their
underlying rationale will be outlined in this chapter. The current study followed Giorgis
descriptive phenomenological psychological method (Giorgi & Giorgi, 2003), which was
described in Chapter Three. This chapter begins by outlining the selection processes adopted
for check captain participants, and then reviews the ethical issues associated with the research.
Prior to the main study, trial interviews were undertaken to evaluate the effectiveness of the
data collection procedures. As the main data collection instrument was the interview, the
chapter deals with the interview settings, together with data collection techniques. A review of
how the data was analysed will also be outlined, and ways in which research conclusions were
justified. The chapter concludes by reviewing how the findings were presented to
representatives of the aviation industry in the form of informal discussions, conference
presentation, and field testing in an international airline.

4.2

Participant selection

Research participants were chosen strategically from target groups, as tends to be the case in
phenomenological research, where the focus is firmly based on descriptions of the life-world,
and sample size tends to be small and concentrated (Smith & Osborn, 2008). Research with
one participant, although not ideal, delivers enough data for findings to be made, and there are
74

Chapter Four
published studies where useful findings have been elicited from one participant (Smith &
Osborn, 2008). As participant numbers increase, findings become richer and more descriptive
(Giorgi & Giorgi, 2008). One particular challenge of qualitative studies, such as those
following a phenomenological approach, is that they can be extremely time consuming (Gay &
Airasian, 2000; Smith & Osborn, 2008). Smith and Osborn (2008) suggest committing to
detailed analysis of fewer participants, rather than scant and broad descriptions of larger
numbers. Hence, they noted a current trend towards decreasing sample sizes. For instance,
numerous phenomenological studies use small sample sizes (numbers between 5 and 9), in
fields such as training, development and standards (Black, 2009; Camp, 2007). Additionally,
qualitative studies in aviation research are also employing small sample sizes. For example,
Henderson (2009) used five crews to observe ten flights (two each crew) for research into
situational awareness, and Lempereur and Lauri (2006) used seven airline pilots in their
phenomenological study into the psychological effects of constant evaluation. It has been
suggested that five to six participants is adequate and appropriate for research of this nature
(Smith & Osborn, 2008).

Given the target group required for the research at hand was experienced check captains, a
deliberate and focused approach to sampling was used, called purposive sampling by
qualitative researchers (Cohen, Manion, & Morrison, 2000). It refers to researchers seeking a
select group of participants who are appropriate to the research. Gay and Airasian (2000)
identify two types of purposive sampling, homogenous and criterion sampling. Homogeneous
sampling refers to the selection of participants with similar outlooks, experiences and
perspectives. Closely related is criterion sampling, which uses established criteria that are
relevant to the research, for example, years of service (Gay & Airasian, 2000). In this case, the
criterion approach was better suited for the study, given the specific criteria, for sampling such

75

Chapter Four
as experience in assessing pilots. Hence, the method adopted was a purposive, criterion-based
selection.

In selecting check captains, it was appropriate that a level of experience and expertise in the
role of check captain was required. The concept of expertise as judgment and appropriate
practice refers to an understanding of what is appropriate at the time, and the ability to execute
the appropriate action effectively (Stevenson, 2003, p. 18). Experience over time was required
for check captains to develop rich understandings of variations in standards. Consequently, it
would be expected that more experienced check captains would have been exposed to a larger
number of assessments of varying standards. The research literature suggested 10 years
experience may be required to develop expertise within particular vocations (Hayes, 1989;
Lave, 1988). Given the high level of expertise that is required within the role of check captain,
it was prudent that 10 years experience in the role of check captain was the minimum
necessary for participation in this research. In the current study, five check captains were
interviewed; their years of experience in the role of check captain was 19 years, 28 years, 17
years, 29 years, and 28 years. This gave a mean of 24.2 years experience in the role of airline
check captain. Five participants were involved (referred to as CC1, CC2, CC3, CC4, and CC5),
with two additional check captains used in the trial interviews (see below). The participants
were sought through industry contacts.

4.3

Ethical issues

It is critical to protect the identity of participants and companies both during and after a
research project (Cohen et al., 2000). As a result, certain protocols formed part of the research
methodology. For ethical reasons, the rights of participants were respected and protected
through informed consent. This was accomplished by addressing four elements: competence,
76

Chapter Four
voluntarism, comprehension, and full information (Cohen & Manion, 1994). Competence
suggests participants make informed decisions before participating, if information provided is
clear and relevant. Voluntarism ensures participation is freely given. Full information implies a
fair explanation of the purpose of the research and expectations of interviewees, and an
understanding that consent may be withdrawn at any time. Comprehension involves
understanding the nature of the research. If all four elements are complied with, then the rights
of individuals have been given suitable consideration (Cohen & Manion, 1994).

To comply with the above protocols, initial telephone contact with check captains entailed an
outline of the research purpose. When check captains agreed to be involved in the study,
university-approved participant information sheets and consent forms were sent to each check
captain (Appendix 3). Once they had been signed and returned by the check captain, an
interview time was scheduled. Additionally, prior to commencement of the interview, each
check captain was again given an overview of the purpose to the research, and told that they
may withdraw from the interview at any time. It must be noted that no check captain refused to
be involved, or decided to withdraw from the study during the interview.

The security and integrity of verbal recordings, transcripts, and field notes became an
important consideration once the interviews had been completed. Data gathered during this
research project was of a sensitive nature, and in all interviews, names of individuals and
airlines were used openly by the participant. This information was subsequently removed to
de-identify it from the transcripts, thus making personal details unidentifiable. As such, all
reporting of research findings involves the release of unidentifiable information only.
Furthermore, all recordings are locked in a secured cabinet on university grounds.

77

Chapter Four

4.4

Trial interviews

In order to ensure high quality data, it was important to pre-test the interview schedule,
recordings, and overall interview procedure. As Gay and Airasian (2000) suggest, the fieldtesting of the research plan in whole or in part is like a dress-rehearsal. They state that trialing
data collection techniques for beginning researchers may provide improved results.
Additionally, Foddy (1995) reports that, after interviews, questions can be re-examined. Two
trial interviews were conducted in the participants own homes. The interviews allowed
reflections on interviewing technique and questions asked. During the trial interviews, an issue
arose around a question relating to minimum standards required in performance by captains.
Both check captains interviewed had difficulty in accepting that they allowed candidates to be
passed if they reached a minimum standard. The main study changed minimum standards to a
focus on assessments that in the past were difficult to alleviate the issue around minimum
standards.

4.5

Data collection and analysis

In the current study, the descriptive phenomenological psychological methodology of Giorgi

and Giorgi (2003, p. 243) was used, and has been outlined in the previous chapter. Proceeding
steps were followed:
1. Obtaining check captain descriptions via semistructured interviews;
2. Transcription of interviews;
3. Using reduction to bracket the researchers experiences;
4. Initial analysis by reading of interview transcripts;
5. Dividing the interview into parts;
6. Ordering the interview into disciplinary language; and
78

Chapter Four
7. Expressing the structure for assessing pilots performance;
Each of these steps is described below.

4.5.1 Obtaining check captain descriptions via semi structured interviews

Interviews conducted with each of the five check captains lasted between 60 and 120 minutes.
At their request, three of the five interviews were conducted in participants own homes (CC1,
CC3, CC4), with the remaining two interviews (CC2 & CC5) conducted at the participants
place of employment. One interview which commenced at the check captains place of
employment (CC5) was subsequently moved to a coffee shop, as the room being used had been
double-booked for an interview. This did not appear to bias the interview in any way, although
final recordings were more difficult to transcribe, given the increased noise. Three airlines
were represented by these five check captains. Three of the check captains resided in the
researchers home state and were interviewed there, while the remaining two were interviewed
interstate. In previous employment, the researcher had limited professional contact with four of
the check captains.

All interviews followed a semistructured interview format. As can be seen in Appendix 4, the
interview schedule had a format that provided direction. All interviews began the same way, by
the researcher asking check captains to discuss assessment of a candidate being considered for
promotion to captain (or, using aviation jargon, command upgrade). So as to develop as much
depth and richness as possible, the interviewer attempted to cover as many assessments as
possible in which the check captains considered the applicants performance to fall around a
pass/fail performance level (referred to as minimum standards in earlier interviews). Further,
as the initial assessment of a candidate occurred in a simulator, and the final assessment took
place in an aircraft, both were discussed.

79

Chapter Four
All interviewees were relaxed, with some interviews jumping ahead without prompting into
areas scheduled for discussion later in the interview. As a consequence, the interview schedule
was used rather like a checklist to ensure the researcher had covered all areas. The approach of
allowing the interview to go in various directions maintained the natural flow of the
conversation. All check captains quickly realised the researcher was asking for concrete
examples of actual assessment, rather than their reasoning behind them. Additionally, it was
made apparent early in all interviews that, even though the researcher was an airline captain,
descriptions were to assume the researcher had limited aviation experience. This request was
made to prompt the check captain to provide as much detail as possible for the purpose of data
analysis.

During the interview, the researcher took notes about areas of interest. This allowed for follow
up questions to be noted and be asked at a later time. Throughout all of the interviews, a small
recorder was placed between the check captain and the researcher to digitally record the
interview. The use of the recording device did not appear to impede the flow of the interview,
or the quality of the information disclosed by the interviewees.

4.5.2 Transcription of the interviews

All interviews were transcribed verbatim, with careful checking to ensure accuracy of the
transcription. As outlined in 4.3, it was important to remove any identifying information. As
mentioned, check captains openly discussed their airlines names, routes, airports into which
they flew, and aircraft type. In some cases, names of individuals who had failed assessment
were discussed. All text that could identify any of the above was removed and replaced with
more generic statements (see Table 7). This would be represented in the transcripts by square
brackets [text removed]. All of the translations can be seen in the following table. Additionally,

80

Chapter Four
some of the transcripts in Chapter Five, have words added, contained between square brackets,
to assist the reader in understanding the context of the interview. For example [words added].

Table 7 Table of common terms used in transcripts

Statement from interview

Transcribed statement

Larger aircraft with two isles such as Boeing 747, 767, Airbus 330

[Wide Body]

Smaller jet aircraft like Boeing 737, Airbus 320, Embraer 170/190.

[Narrow Body]

Smaller turbo prop aircraft like F27, F50, Saab 340

[Turboprop]

Capital City like Sydney, Perth

[Capital City Airport]

Larger city like Gold Coast, Townsville

[Large Airport]

Smaller regional airports like Launceston, Mackay

[Regional Airport]

General comments i.e. names or airlines

[name removed]

4.5.3 Using reduction to bracket the researchers experience

As outlined in the previous chapter, the researcher during the analysis must practise
phenomenological reduction, by bracketing preconceptions and assumptions as much as
possible (Giorgi & Giorgi, 2003; Hein & Austin, 2001). The researcher has flown aircraft of
various sizes for over 27 years. He has been involved in the training of pilots on jet aircraft, as
well as developing and introducing training programs for both training and assessing pilots on
domestic and international airlines. In setting aside assumptions based on this experience, the
reduction requires the researcher to remain open to the data. Bracketing the researchers
experiences made it easier to analyse the data, by requiring the researcher to focus only on the
evidence presented.

81

Chapter Four
4.5.4 Initial analysis by reading of interview transcripts
Prior to analysing the data, the researcher developed a level of familiarity with each interview
by reading the transcripts. This helped the researcher to extend his understandings of the
experiences of each check captain during the assessment of pilots undergoing promotion to
airline captain. A level of familiarity was also gained during the checking of the interview
recordings against the transcripts. Even so, after each transcript was completed, it was read
again prior to moving on to the next stage of analysis. This process took, on average, about 15
hours for each interview.

4.5.5 Dividing the interview into parts

As described in Chapter Three, meaning units may be single words, sentences, or paragraphs,
in their original form spoken by participants. After reading each transcript, the researcher
marked each point when a change in meaning occurred, thus creating a meaning unit. In many
cases, within each meaning unit, both the researcher and check captain were speaking. To
differentiate the individual speaking, each time the conversation changed between the
researcher or check captain, the numbering changed. For example:
CC3: Yes, ah, in which area are you wanting me to go down. (CC3:3)
This describes CC3 talking in his interview, and would describe the third turn in the whole
interview. By referring to Appendix 5, illustrates to the reader how the numbering works, and
how meaning units evolved.

4.5.6 Transforming the interview into disciplinary language

As interview transcripts are generated from conversations about the research topic, it was
necessary to transform them into the disciplinary language of the study. After each meaning
unit was created, it was transformed into disciplinary language. For each of the five interviews,
three transformations were carried out, as illustrated in Figure 12. Transformation 1 (TR1) was
82

Chapter Four
a summary of each meaning unit within an interview transcript. Once TR1 had been completed
and numbered, similarities between units within TR1 were identified. They were combined and
further summarised as TR2. This process of grouping similar units was continued, creating
TR3. In TR3 however, rather than numbering, each component was summarised under a key
heading, such as situational awareness, flying skills, decision making. Appendix 5 shows an
example of meaning unit formation and transformations into disciplinary language. A full
analysis was completed for CC1 prior to moving on to CC2. Once the process became familiar,
the analysis moved more quickly for each check captain, with the analysis for CC1 taking
around 100 hours while the analysis for CC5, was completed in around 40-50 hours.

Figure 12 Illustration of analysis of one interview transcript for one check captain

4.5.7 Expressing the structure for assessing pilots performance

Developing a structure from the data, involved writing a single paragraph expressing the
criteria used to assess pilots undergoing promotion to captain. To assist in developing this

83

Chapter Four
structure, all TR3s from each check captain were transferred into a single table (see Appendix
6), with Figure 13 illustrating how this occurred. Additionally, trying to develop a visual model
assisted in the development of the structure. Once the model held together, and was consistent
with the transcripts, it enabled the researcher to write the structure.

Figure 13 Combining all five check captains TR 3 to develop the final structure.

4.6

Justification of findings

Chapter Three argued that although validity and reliability have a strong focus in research
under pinned by positivistic theory; this can also be the case in interpretive research, so that it
can also produce valid and reliable results. This section relays how validity and reliability
issues were considered.

84

Chapter Four
4.6.1 How validity was assured
Chapter Three (3.5.1) highlighted the need for interpretive research to fulfill communicative,
pragmatic, and transgressive validity requirements. Methods of meeting each of these
requirements will now be outlined, including how the research findings were communicated to
the wider aviation community.

4.6.1.1 Communicative validity

In the current study, communicative validity was achieved by acquiring and validating
knowledge, taken from participants during interviews (Kvale, 1995, 1999). Communicative
validity requires three steps. First, it assumes that the researcher and participant understand the
purpose of the research (Sandberg, 2005). This was accomplished in a number of ways. During
initial contact with the check captains, the research aim was described over the phone. Once a
check captain had agreed to be involved, a participant information sheet was mailed to the
check captain describing the research (see Appendix 3). At the beginning of the interview with
each of the check captains, the intention of the research was again described. Additionally,
during interviews, the dialogue between the researcher and the check captain centred on
minimum standards or pass/fail assessments. For example, whilst interviewing, the researcher
constantly attempted to gain a vivid picture, by way of check captain descriptions, about
particular candidates who were performing at around a pass/fail standard. For example, the
statement below is the researcher asking CC3 a question about minimum standards.
Have you done one in the past where a decision to pass or fail has been a difficult one
where you have just had to sit there and scratch your head just a little bit? (CC3:14)
Another way was asking the check captains what and how rather than why questions. To
do this the researcher asked questions like:

85

Chapter Four
R:

can we now focus around that point where you picked up that all of a sudden

somethings going on. Can you go through that and just sequence what you were
seeing. (CC1:139)

R:

Can you give me an example of what he did to be antagonistic? (CC2: 29)

The second component of communicative validity requires the researcher to preserve

coherence (Sandberg, 2005). This is achieved when there is logic and consistency in an
argument (Kvale, 1995; Sandberg, 2005). Coherence was achieved by initially interpreting the
individual check captains descriptions of minimum standards, as outlined in Section 4.5.6.
Once each check captains interviews had been interpreted, the entire check captains
description were brought together as a whole (assisted by referring to the combined final
transformation TR3 of all five check captains, see Appendix 6). By viewing interpretations
individually and collectively, it enabled the researcher to improve the coherence of the
researchers interpretations.

Third, communicative validity is achieved by discussing findings with participants and other
researchers (Sandberg, 2005). On completion of the analysis, a strategy of discussing the
findings with other researchers and pilots was used. This strategy will be taken up further in
Section 4.6.1.4.

4.6.1.2 Pragmatic validity

Communicative validity facilitates ways of improving interpretations of interviews. Pragmatic
validity, on the other hand, is confirming that individuals are in fact doing what they say they
are doing (Sandberg, 2005). Sandberg outlines that by asking follow up questions assists in

86

Chapter Four
developing deeper descriptions, and confirms that what is being discussed, is, in fact, true. The
following is an example of a follow up question asked by the researcher
Yep. So if I can get back to this person before, where you said his decision making,
was it decision making that you were a bit ... (CC3:29)
Sandberg (2005) also supports using deliberate misinterpretation as a way of checking
pragmatic validity. For example, during interviews, questions can be asked, that incorrectly
quotes or misinterprets the participants previous descriptions. When they correct the
researcher, and reiterate their initial statements, this improves pragmatic validity. This was
used during the trial interviews; however, it was not used during the main study for one main
reason. Making deliberate misinterpretations during interviews, actually made the researcher
feel extremely uncomfortable. During the trial interviews, one check captain appeared to sense
that the researcher was trying to misinterpret their descriptions.

Pragmatic validity can also be proved by validating interpretations and findings, and using
them in practice (Sandberg, 2005). The current study was able to take the research findings into
the wider aviation community and test the findings. As with communicative validity, this will
be discussed in Section 4.6.1.4.

4.6.1.3 Transgressive validity

In the current study, communicative and pragmatic validity provided a means of improving
ways of obtaining knowledge via interviews, and confirming if this knowledge was what was
occurring in practice (Kvale, 2001; Sandberg, 2005). Transgressive validity assists the
researcher in trying to become aware of their taken-for-granted frameworks (Sandberg, 2005,
p. 58). Even if a researcher does not find disagreements, tension, and uncertainty which might
occur within the interpretations, transgressive validity is achieved if the researcher can
demonstrate they actively searched for it. A researcher, after initial interpretations of lived
87

Chapter Four
experiences are made, decisively attempts a reinterpretation of the data (Sandberg, 2005). This
occurs through the process of analysis, and form of free imaginative variation (Sandberg,
2005, p. 61). In other words, can comments by check captains be seen other ways? Throughout
the research, the transcripts were read and re-read to see if differing interpretations could be
made. For example, through the final analysis, the researcher found himself going back to TR3
for a check captain, and then tracing back to TR2, and then to TR1, and then back to the
original interview. What the researcher was searching for were ways of actually reinterpreting
the initial analysis of the interview. An example was during the analysis of CC1 interview. He
made a comment about a candidates loss of situation awareness by the comment:
Do something that tells me that you know where you are. Do something, I am saying to
myself. But he didnt! (CC1: 140)
This comment by CC1 captured the importance of situational awareness that many of the check
captains had made. However, in the latter part of analysis, the researcher went back into
comments like that of CC1, and asked whether situational awareness was in fact what the
check captains were really discussing. In this situation, no other interpretations could be made.

As was outlined in Chapter Three, Sandberg (2005) argues that the weaknesses of
communicative, pragmatic and transgressive validity are reduced by the strengths of the others.

4.6.1.4 Spreading the news communicative and pragmatic validity

As outlined in 3.5.1.1, Sandberg (2005) reflected that one component of communicative
validity authenticates research findings, by discussing findings with participants and the wider
research community. Additionally, as outlined in 3.5.1.2, pragmatic validity can be achieved
by testing interpretations and findings in practice. On completion of the data analysis, several
steps were undertaken to fulfill these obligation. First, discussions with additional check
captains occurred once a model for assessing pilots performance (MAPP) had been developed
88

Chapter Four
from this study (see Section 5.3). A relatively junior check captain who had been assessing for
two years took part in these discussions. He described how only recently, he had failed a
candidate who was being assessed for promotion to airline captain, due to what the check
captain described as loss of SA, during a flight into a regional airport at night. The check
captain described how anxious he was, conducting this assessment, given that no technical
limitations had been broken. With the use of the MAPP, his assigning of a failure grade was
justified. After looking at the model, he gave what is called the phenomenological nod of
recognition.

Second, the researcher discussed the findings with the five check captains in the main study. A
phone call was made to each one discussing the structure, and the model that was developed.
All the check captains agreed that the findings corresponded to their method of assessment.
Two in particular were impressed by how simple, yet complete, the model was.

Third, presentations were made at an aviation education and research conference in New
Zealand (Mavin & Dall'Alba, 2009) and also at a international aviation psychology conference
in Sydney (Mavin & Dall'Alba, 2010). One airline pilot attendee at the conference in New
Zealand asked if he could use the PowerPoint presentation in order to discuss the model with
check captains during a course that was being conducted soon after. He emailed soon after
with:
There was a considerable amount of agreement with your findings and a lot of interest
in your work. We had a very good discussion for some time and while nobody had
specifically categorised the critical factors for themselves before, the gut feeling
issue of pass/fail on NTS was generally in line with your thoughts on the critical and
enabling skills concept, with SA and decision making being higher order requisites. I

89

Chapter Four
still haven't found anybody who disagrees with your research which is a good thing.
Hope I can say the same thing in five years time!
In addition to the remarks made above, comments were made by an air traffic control
educator who recognised a strong link between the findings and what he saw during training of
air traffic control students.

The final way in which the findings were evaluated was a 5-day check and training workshop
that was run by the researcher. The purpose of this workshop was to develop instructional and
assessment techniques for new and experienced check and training captains in an international
airline. As five of the seven individuals had previously been involved in airline pilot
assessment, it was a good opportunity to discuss and field test the model. During the three days
of assessment training, the model was used as a later backdrop for assessing five out of eight
videoed flight scenarios. Of the eight scenarios, there were three that would be considered to
show performance at or around a minimum standard.

To begin the training session, each pilot was given instructions on the technical skills and NTS
criteria (see in Appendix 1 and 2). When the pilots were asked to assess the first three videoed
scenarios, there was a level of inconsistency occurring between overall grades given. For
example, even though the pilots agreed that the performance in the video was marginal, some
of the eight were unable to explain why, or recognised only the technical errors. Additionally,
there was a degree of variability between the experienced pilots, in regard to the performance
being either satisfactory or unsatisfactory.

When the MAPP was introduced for the remaining five scenarios, it assisted all individuals to
determine whether pilot performance in a particular scenario was acceptable or required retesting. Furthermore, the criteria used to assess performance of the videos, became more
90

Chapter Four
consistent between the eight pilots. Additionally, the MAPP encouraged the pilots to look at
other areas, rather than just technical skills. More importantly, the five experienced check
captains in the group were comfortable that the model supported the judgments that they would
normally make.

4.6.2 How reliable results were achieved and demonstrated

Interpretative research argues (see Section 3.5.1) that reliability occurs through the repeated
and open justification of the rationale behind the researchers actions. As outlined in Chapter
Three, research projects, such as the current study, generally take into account the six main
areas listed below in order to describe how reliable results can be achieved and demonstrated.
The areas were:
1.

How interview questions are devised;

2.

Outline of how and why participants were selected;

3.

A detail of procedures and processes used for data collection;

4.

How data is analysed;

5.

Provide sufficient detail and examples of how results were obtained; and

6.

Provide a paper trail for other researchers wishing to confirm or audit research.

This chapter has described in detail the requirements for points one to four outlined above. The
following chapter will present adequate information and direct quotes from the five check
captains regarding the way in which the results were determined (Point 5). It is important to
note that it was impractical to present the vast quantity of data and analysis this research
produced (around 100,000 words). As a result, the researcher presented representative
samplings to substantiate the findings. However, all details of the entire research project,
including interview recordings, notes taken during all interviews, transcripts, and all
transformations of each of the five check captains have been retained. Consequently, the
researcher has satisfied the requirements for Point 6, by providing a paper trail.
91

Chapter Four

4.7

Summary

This chapter has provided a synopsis of how this research was carried out. It has outlined
participant involvement, ethical considerations, as well as the trial interviews that were
conducted. Additionally, the steps taken in the main study were outlined. This chapter has also
examined the way in which valid and reliable research was conducted. Moreover, the chapter
reviewed methods for achieving communicative and pragmatic validity by discussing the
results with the wider aviation community.

92

Chapter Five

Chapter Five
Findings
I was going to say to him Look, I sat there with my fingers
crossed saying, youve done such a good job up until now, please
pull up your data, please. Do something that tells me that you know
where you are. Do something, I am saying to myself. But he
didnt! (CC1: 140)

5.1

Introduction

The aim of the current study was to establish the criteria used by experienced check captains to
assess airline pilots performance. In particular, the research focused on identifying the actual
criteria used to determine whether a pilot had achieved at least a minimum standard during
promotion to airline captain. The research interviews focused on gaining descriptions of
assessments where pilots had performed at or around this minimum standard. This chapter
describes the findings from the analysis of these interviews. It begins by providing a synopsis
of each check captains responses during their interviews. A structure is then developed, and
provided in a narrative form, outlining the constituents of this structure: SA, decision making,
flying skills, aviation knowledge, management, and communication, with examples provided.
From this structure, a model of assessing pilots performance was developed. The chapter
concludes by providing an extensive rationale behind each of the constituents.

93

Chapter Five

5.2

Summary of pilot assessment for each check captain

During each interview, there was a focus on describing examples of the assessments that each
check captain had conducted; those where the performance of the candidate was around a
minimum standard. This approach was to determine the criteria, or in phenomenological terms,
the constituents, used to measure pilots performance. The research approach was not to
identify a minimum standard, but to determine the constituents used in current practice to
assess pilots performance. The following is an overview of each of the five check captains
descriptions of constituents used to assess the performance of pilots undergoing promotion to
airline captain.

5.2.1 Check Captain 1 (CC1)

There appeared to be a real willingness of CC1 to describe several separate situations where
the decision to pass or fail candidates were difficult, or the candidates performance was
ambiguous in terms of what CC1 considered a minimum standard. Various themes emerged in
these descriptions, with the most important for him being that of SA. CC1 described SA as
obtaining information in various forms, such as from navigational systems within the aircraft,
or from outside the aircraft, for example, other aircraft or terrain. Other themes such as
decision making, communication, and understanding standard operating procedures, were
described as important. A theme that was mentioned almost in passing was that of flying skills,
with CC1 making vague assumptions that candidates must be able to fly the aircraft within
specific tolerances laid down by the regulatory authority.

5.2.2 Check Captain 2 (CC2)

CC2 relayed various assessment scenarios that effectively described minimum standards. He
considered it to be important that candidates acquired information from sources inside and
94

Chapter Five
outside the aircraft, so that sound decisions could be made and SA could be maintained. For
CC2, decision making and SA were extremely important, with demonstrations of both being
required under difficult assessment conditions. However, CC2 did note that SA was difficult to
assess. In relation to assessing SA, he gave examples of assessments where individuals had
displayed a loss of SA, rather than describing how he measured a candidates SA. CC2
described the ability to fly the aircraft as extremely important. Despite the fact that flying skills
were crucial, minor digressions in flying skills were excused if SA was maintained. Other
themes that emerged were the importance of management and leadership skills, understanding
standard operating procedures, self assessment skills, and cooperation skills.

5.2.3 Check Captain 3 (CC3)

CC3 outlined several important areas for candidates during promotion to airline captain. He
outlined assessments during which candidates had not taken into account information he
considered important, such as navigational equipment whilst airborne, engineers whereabouts
relative to the aircraft during ground operations, and monitoring the flight path of the aircraft
via aircraft instruments. For CC3, the ability to source such information was essential to the
development of SA. Candidates unable to do so were considered to be below a minimum
performance standard. CC3 described candidates needing to prioritise information for effective
decisions to be made. Importantly, during the decision process, he thought that candidates
should display confidence, and be consistent without continual modifications of decisions. CC3
described that candidates must be able to fly the aircraft to a reasonable standard and operate to
company standard operating procedures.

5.2.4 Check Captain 4 (CC4)

While CC4 outlined similar themes to the previous three check captains, interestingly, new
themes also emerged. CC4 outlined that candidates must operate aircraft in ways that allow
95

Chapter Five
them to maintain SA at all times. Coordination and prioritising of individual and whole crew
activities increased the ability of candidates to obtain information necessary for developing SA.
As with the previous check captains, CC4 believed that flying the aircraft within tolerances laid
down by the company or regulatory authorities was important. He also outlined that candidates
must have some level of technical knowledge of the aircraft, systems and other aviation related
concepts.

5.2.5 Check Captain 5 (CC5)

CC5 outlined numerous assessments of candidates that he considered marginal. He was more
descriptive in linking themes already outlined by the other four check captains. He stated that
candidates must be capable of gathering information from various sources inside and outside
the aircraft in order to develop SA, and make decisions. He described situations where
candidates SA should have been shared with other crew members. CC5 outlined that the
inclusion of other crew members in decision making is important. Another issue in relation to
decisions was the ability to balance the risk of differing alternatives. For CC5, managing the
operation so that SA could be maintained, and the aircraft kept within regulatory parameters,
was fundamental. For him, communication between other crew members, including first officer
(FO) and cabin crew was a way of receiving further information needed for SA and decision
making. It was also seen as a way to effectively direct crew once a decision was made.
Interestingly, CC5 stated that poor decisions or reduced SA may manifest as aircraft not being
flown smoothly, or within company or regulatory limits.

5.3

Criteria used to assess pilots performance

From the above five check captains, a final structure was developed, describing the
constituents (criteria) used to assess pilots performance undergoing promotion to airline
96

Chapter Five
captain. The structure of assessing pilots performance, that is, the constituents used and their
relationships, is described in the following paragraph.

During flight operations, candidates must continually obtain and prioritise information from
the environment, including information about the activities of other crew members, for the
purpose of maintaining SA at all times. Aviation general knowledge is necessary for
understanding and prioritising this information. Facilitating effective flow of information
amongst crew assists in maintaining and improving SA of the candidate and other crew
members. SA is necessary for making decisions and taking action which takes risk into
account. Improved decisions can be achieved by facilitating and encouraging inputs from other
crew. Candidates must ensure decisions are understood by crew members whose current and/or
future activities are affected by these decisions. When crew activities are not occurring
efficiently, or are not in alignment with company standard operating procedures, candidates are
required to actively prioritise crew routines and specialised procedures. This prioritisation
enables the aircraft to be operated manually or with automation within company or regulatory
specified standards, while efficient and effective crew activities are maintained within
company standard operating procedures.

The structure above is made up of six constituents: SA, decision making, flying skills, aviation
knowledge, management of crew activities, and communication amongst crew. This structure
is holistic in nature, with the relationship of the constituents as central as the constituents
themselves. Figure 14, a model for assessing pilots performance (MAPP), highlights that
performance comprises both essential and enabling constituents. Essential constituents are
those on which the check captains placed emphasis. A significant drop in any of these areas
would indicate that the pilot was below a minimum standard. Enabling constituents assisted in
maintaining or improving their essential counterparts.
97

Chapter Five
In developing the structure from the research interviews, each attempt to remove any of the
enabling constituents from the structure resulted in the structure collapsing. This implies that
the constituents form a hierarchical, dynamic structure, as shown in Figure 14.

Situational
Awareness

Essential

Enabling

Aircraft Flown
Within Tolerances

Aviation
Knowledge

Decisions
Considerate of Risk

Management
of Crew

Communication
Amongst Crew

Figure 14 Model for assessing pilots performance (MAPP)

5.4

Rationale and supporting evidence for the structure

As outlined in Section 4.6.2, phenomenological research requires researchers to show not only
how constituents evolved, but also how they are related in forming the structure of a
phenomenon (Giorgi & Giorgi, 2008). This section does two things: first, it describes how each
constituent was developed from the transcripts of interviews with experienced check captains;
and second, it examines the way in which each of the constituents is related to the others.

98

Chapter Five
5.4.1 Situational awareness
Each check captain articulated the significance of candidates displaying evidence of SA during
assessment: SA for me is, probably, is the most important thing (CC2: 15). SA was expected to
be demonstrated by individual candidates at all times, with lapses in SA creating concerns. For
example, CC1 described an assessment of a candidate who was flying into an airport that was
generally more difficult than others. The difficulty stemmed from air traffic control giving
unusual navigational requirements. Up until that point, the candidate had performed well.
However, due to these specific requirements, he was required to navigate via a radio navigation
aid, rather than the inertial navigation system1. At that time, the radio aid would give a higher
degree of accuracy, which was what was needed to maintain the aircraft within the required
navigational tolerances. The candidate had forgotten this requirement, although the FO he was
flying with had not. The candidate was failing to recognise prompts from the FO about this
requirement. Although there was a brief moment of confusion, the candidate recovered and
maintained the aircraft within the requirements laid down by the regulatory authority, though
doubt was raised about his level of SA.
R:

What was the one issue that was concerning you, what was it if you can use
your words again?

CC1:

The situational awareness.

R:

SA again was the thing that was concerning you? Had he displayed any SA
problems in the past?

CC1:

No, none at all.

R:

Okay, so it was just an SA thing again.

The inertial navigation systems, is an internal navigational source, based on laser gyros which, on older aircraft

type, were susceptible to drift error. They were accurate for en-route navigation, though as the aircraft came close
to terrain and air traffic control boundaries, more accurate navigational sources were required.

99

Chapter Five
CC1:

It was just one of those its an SA thing but it was a one off (R: ok) and it was
uncharacteristic of him. (CC1: 132-137)

This event was the only time that this candidate had shown signs of lost SA. The candidate was
subsequently passed, but there was a level of concern about this decision. CC1 sought advice
from another more experienced colleague who disagreed with the decision. This misgiving in
passing the candidate drew attention to the fact that maintaining SA was an essential
component in minimum standards of command.

During assessments, the check captains expected candidates to obtain information from various
sources, including other crew, and to be able to prioritise this information to develop SA.
Outlined below is the way in which candidates must obtain and prioritise information to
develop SA, as well as use communication to maintain and improve SA.

5.4.1.1 Obtaining and prioritising information

During certain phases of flight, differing sources of information became more important at
specific times. The check captains expect candidates to actively source appropriate and
accurate information so they can develop SA and maintain the aircraft within required
parameters. The appropriateness of this information differs, depending on the phase of flight.
In the following example, the candidate is navigating the aircraft by means of a less accurate
navigational system. However, at that point in the operation, the candidate should have been
aware that a more accurate and appropriate system called a VOR2 was available (sometimes
referred to as raw data).

A VOR is a ground based navigation aid. It is split into 360 separate radials, with 000 being aligned to magnetic

north, and other radials similar to a compass rose. Equipment in the aircraft can identify which radial the aircraft is
on, enabling pilots to accurately navigate the aircraft. A general navigational tolerance for a VOR is 5.

100

Chapter Five
From memory, the first time I picked up on it was when I noticed he didnt have the
raw data he didnt have the supporting hard evidence from the VOR (R: mm hm),
the navigation aid to tell him where he was in regard with, relative to the track that he
was expected to intercept. He was just relying on the FMC of the aircraft, the flight
management computer, which can be inaccurate, to do it for him and give him a little
nudge from behind. He was relying on that completely and hed missed the, ah, basic
tenet that you have with supporting your data up from the navaid itself as a support
(R: ok) to confirm whether or not you are actually on the track that youre supposed
to be on (R: ok). He didnt have that up (R: yep). (CC1: 142)
As this example shows, SA came from the candidate monitoring and, when required, switching
to more accurate and relevant sources of information. For CC1, the candidates failure to use
the optimal source of information placed the assessment result as marginal.

When candidates obtain accurate and relevant information, they must prioritise this information
to develop a realistic awareness of the environment. In the following example, a candidate,
during a simulator assessment, had been given a fire in the galley of the aircraft. By CC3
pretending to be a female cabin crew member, the candidate would be able to as realistically as
possible, determine the conditions in the rear of the aircraft. In this situation, the fire was
extremely dangerous, as the cabin crew (CC3) were notifying the candidate that they had used
four fire extinguisher bottles, in an attempt to extinguish the fire. However, at the time, the
candidate was also concentrating on maintaining the aircraft within the parameters directed
from air traffic control (CC3 was also acting as air traffic control).
a guy had a fire in the rear galley (R: okay) and the girls [CC3] were giving him,
ah, prompts with Just fired the fourth bottle into this thing and it was still going. (R:
yep). That should be a real flag to the guy (R: mm), to get this thing on the ground
pronto because we have got an uncontrollable fire! (R: Yep). But he allowed air
101

Chapter Five
traffic control to push him all around the place and he was even going a lot slower
than he should have been anyway before the approach and ah with no urgency and no
forward planning. (CC3: 49)
This shows how a candidate was sourcing information via communication with cabin crew.
Although the candidate perceived the information, he didnt consider the situation evolving in
the rear of the aircraft as critical. In this example, CC3 expected the candidate to consider the
situation as significant and demonstrate a level of urgency by not allowing air traffic control to
delay the landing. This candidate was considered to be below standard.

5.4.1.2 Using information to develop SA

Once candidates source and prioritise accurate and relevant information for that phase of flight,
the check captains expect candidates to operate the aircraft, and act in ways that demonstrate
this awareness. In the following example, CC3 goes further by outlining the relationship
between flying the aircraft and demonstrating SA. He describes the importance of flying,
though maintains that candidates must be able to operate the aircraft, so that it creates no doubt
for him that they are aware of the important and relevant information, at that time, in the flight.
CC3: Well yeah, if you can fly the aeroplane (R: yeah) there is no problem with the
manipulation side of it, but if he is not really aware that there is high terrain
there and there is three or four aeroplanes coming into this area (R: yep) um,
and the fact that he might be able to fly doesn't mean diddly-squat (R: yep) if
he's going to run into something.
R:

Yeah. So for you, obviously situational awareness appears to be ...

CC3: Yeah very high. (CC3: 270-272)

102

Chapter Five
5.4.1.3 Communicating to share and improve SA
An additional source of information for developing SA comes from other crew members. This
source of information is based on candidates ability to facilitate a flow of information between
themselves and other crew members. This flow of information was seen as significant in
maintaining and improving SA of candidates and crew. In the example that follows, the
candidate had not prioritised or sourced the most accurate information to develop SA.
Accordingly, the aircraft was at risk of exceeding the required navigational parameters. The FO
was aware the candidate was not fully cognizant of the situation, and prompted the candidate
with new information. Additionally, the candidate was acting in such a way that the FO was
comfortable in providing information. Using communication, the FO was able to improve the
SA of a candidate to a level that enabled him to operate the aircraft within required parameters.
So, the FO said right and he put it up on his instrument so that candidate could see
that hed gone through the radial only a very small amount (R: mm hm). So they
actually picked it up, it was all within tolerance, at no point in time in the process of
arriving at [an unusual airport], at no point did he actually break any parameters or
any requirements. (CC1: 126)

During assessment, SA was seen as essential. To demonstrate SA, candidates must obtain
accurate and relevant information for that stage of flight, and supplement this by
communicating with other crew members to assist in developing SA. This information is then
prioritised to develop a realistic SA model. However, it was recognised that SA was difficult to
assess.

5.4.1.4 The difficulty in assessing SA

While SA was seen as essential, it was also identified as difficult to assess: SA is such an
esoteric thing youre got no chance of nailing that down, its just to have a gut feeling (CC2:
103

Chapter Five
289). Although the idea behind SA can be explained, a difficulty arises in assessing it,
especially when compared with more straightforward criteria such as aircraft knowledge and
manipulation skills.
Yeah. I mean thats probably the simplest thing to assess, the manipulation because
youve got a set of criteria that when you take off and you have an engine failure,
youve got to be within certain tolerances (R: mm). If you fail to do that well its, so
its more easily defined, (R: mm) in a sense. The harder things to define are the things
like the, you know, the level of situation awareness, the flight management, the ability
to communicate well with your co-pilot and with the rest of the team in a sense of air
traffic control and engineers and particularly the cabin supervisor. (CC2: 13)

The check captains identified strongly that they were able to recognise when a candidate was
displaying a loss of SA, rather than trying to determine if a candidate actually had SA. For
example, as can be seen in the opening quote in this chapter, CC1 had recognised that the
candidate was showing signs of lost SA. In summary, the check captains identified SA as
essential during promotion to airline captain, though they gave only descriptions of when a
candidate was displaying reduced SA.

5.4.2 Decision making

The research identified that not only was SA an essential constituent, but also decision making.
During interviews, the check captains described decision making as extremely important for
the duration of the assessment for promotion to airline captain. In all, three main themes
emerged. First, candidates needed to accept the responsibility that came with being a captain.
Second, decisions needed to be sound and considerate of risk. Third, communication needed to
be used in improving and conveying decisions.

104

Chapter Five
5.4.2.1 Taking responsibility for decisions
The first element of the decision making process was having the confidence to be able to make
decisions. One candidate, who had initially been viewed as an individual who would eat his
command (CC2: 290), was later assessed as unable to take the final step in taking
responsibility. Another candidate just hadnt really taken the next step from being a FO to
commanding the situation. He was tending to be a little bit uncertain of some of his decisions
(CC3: 24). This ability to take charge of the situation was the first important step for command:
Ive always believed that flying aeroplanes is nothing more than an exercise in self-confidence
(CC1: 201).

Candidates were required to take complete responsibility for decisions and not delegate that to
another person. A concern arose when a candidate attempted to delegate the final decision to
someone else, so that the candidate could then enact that decision. The preferred situation for
the check captains was that candidates acquire information from various sources, including
other crew and company personnel, in order to make informed decisions. This acceptance of
responsibility was an important element of decision making. For example, during decision
making, the problem to be resolved is sometimes difficult to define. On long flights that require
large amounts of fuel, for instance, aircraft can take off at heavier weights than the weight at
which they are allowed to land. As fuel is burnt off during the flight, the aircraft will become
lighter and subsequently land below maximum landing weight. During flight planning, it is the
captains responsibility to ensure the aircraft will land at the destination below the aircrafts
maximum landing weight. When an aircraft has taken off very heavy, an unscheduled,
immediate return to the departure aerodrome is against regulations. However, it is permitted
under emergency conditions, although there are some situations where the line between an
emergency and a normal situation is not always clear.

105

Chapter Five
In the following example, the candidate had to make a decision about the fine line between
flying the aircraft around after an engine failure to burn off fuel, or land overweight.
... he got the FO to enquire of the company representative in [capital city airport]
whos just an agent, whether he could get a phone patch through to [maintenance
personnel], to get permission off whoever was on duty [maintenance facility] to land
overweight (R: mm ). You could say a conservative thing to do .. Thats nice!
But in terms of, he was really just delegating his responsibility for the safety of that
aeroplane and all its occupants (R: mm) to somebody who may or may not be
qualified to make that decision. (CC2: 34)
This example demonstrates a candidate being unwilling to take responsibility for a decision,
and land overweight. As can be seen, the pilot was attempting to absolve him/herself of the
responsibility, by seeking permission from other company personnel. Whilst CC2 considered
the approach conservative, he deemed this candidate not to have achieved a necessary
minimum standard of performance.

5.4.2.2 Making sound decisions that are considerate of risk

Another key element of the decision process is the candidates ability to weigh the benefits of a
decision against the risks. As with SA, the check captains found risk difficult to assess in pilot
performance. In the following example, an assessment on a real aircraft, the plane is about to
depart an airport that has active thunderstorms, which may have associated large wind changes,
called microburst. These wind changes are very dangerous for jet aircraft and have created
accidents in the past. In this example, CC2 was concerned with the risk associated with taking
off towards an active thunderstorm, which may have had an associated microburst.
But youre really better off going towards the storm, (R: yep) and then you can see the
damn thing and if it does get to active microburst or anything, you know, youre
heading into it. (R: yeah) Its safer than heading away from it. (R: yep) So we had
106

Chapter Five
this, you know, we took off, I was sitting there, I knew they were going to do it but I
thought do I interfere and say No were not going to take off this way? The end
result was I sat there. (R: And crossed your fingers). CC2: And we took offof course
nothing happened on the way there, so we got it there on the ground alright. I was
actually so agitated about it. We got to top of climb, which I was annoyed at myself
for doing, but I said Look, what do you guys think about that takeoff? And both of
them, and to my great disappointment, the trainee check guy reckoned it was all fine.
(CC2: 196-198)
CC2 outlined that his unease may have been supported by around 70-80% of other pilots. For
him, the candidate had demonstrated a marginal performance in assessing the risk associated
with a decision. In this case, it was due to a lack of awareness of risk; the overall assessment
was a pass but was what he described as a wobbly tick (CC2: 203). Once a candidate is able to
take responsibility for a decision, and make decisions in the light of risks, they are able to
improve decisions, and convey their decisions, through communication.

5.4.2.3 Communicating to improve and convey decisions

The check captains identified two important aspects in relation to decision making. First,
decisions can be improved by having input from other crew. Second, once decisions are made,
pertinent components of decisions need to be relayed to relevant crew members. Both of these
elements require the ability to communicate as the medium for information flow.

Having external critique or input was seen to assist candidates to confirm or enhance decisions.
Other crew, especially the FO, is a resource that should be used during the development and
finalising of decisions. Candidates who encourage this sharing of information with other crew
were considered to make better decisions. CC1 described a preferred method of allowing a FO

107

Chapter Five
to critique decisions: Hell accept criticism. Hell say, Oh yes I forgot that. He wont let it
worry him; hell just get on with it (CC1: 206).

The other essential component of the decision process is to inform crew, whose task will be
affected by the decision. In the example below, the candidate made a decision to divert to
another aerodrome, owing to an inability to land at the intended location due to poor weather.
The result was that the aircraft landed at this alternate aerodrome with the required fuel
reserves. However, the decision to divert did not concern CC3; the concern was that key
components of the decision were not relayed to the FO, whose duties would have been affected
by such a decision.
In that case there was no reference to how much fuel he had left. He probably knew
how much fuel he had left, but he didnt mention it to the FO: I have got x amount
of fuel left and we are going to burn this to get to where we are going, (R: yep) so
therefore we are okay. And that was never mentioned. So we diverted and, ah,
landed, and landed with the required reserves, (R: yep) but there had been no mention
of it throughout the whole deal. (CC3: 36)
Specifically, the candidate had not relayed the details of the current fuel and expected fuel
usage during the diversion. In this case, the candidate was seen as a marginal fail.

5.4.2.4 Linking SA and decision making

SA and decision making emerged as essential constituents in assessing pilots performance. In
addition, SA and decision making were seen to be linked. For example, a candidates decision
to divert an aircraft back to an airport was poorly executed when the aircraft flew through the
top of a thunderstorm. This had occurred not once, but twice. When asked whether this was the
result of the decision-making process, CC3 explained: Well, what lead him into that situation

108

Chapter Five
was SA and he didnt have the SA of those thunderstorms (CC3: 299). This example shows that
the poor decision was the result of poor or inadequate SA.

5.4.3 Flying skills

In addition to SA and decision making, flying skills were also considered to be an essential
constituent. Flying skills related to the candidates ability to manage the flight path of the
aircraft within the benchmarks laid down by the regulatory authority or company standards.
This capability could be expected with either automatic pilot or manual flight in normal and
non-normal operations. Given that flying skills have, in some cases, been embedded as the sole
measurement of pilots performance, it was not unexpected that the check captains saw them as
essential constituents. The art of flying an aircraft is essential for, if not rooted in, an individual
pilots tool box of skills. Some of the check captains appeared to assume that the candidates
were able to manipulate the aircraft to within the required company and regulatory standards.
For instance, CC4 saw the ability to fly the aircraft within the designated standards as essential
manipulative ability is simply [specific airlines] standards. You know, thats the minimum
standard (CC4: 28). Comments like this were common with each of the check captains. For
example, CC2 made a statement about flying skills as being an assumed pre-requisite skill.
Youve got to understand that ... and youve obviously got to meet statutory minimum, to meet
the level of ability (CC2: 288).

5.4.3.1 Linking flying with other constituents

Although flying skills were seen as essential, it was outlined by a number of check captains
that poor flying skills are often related to an inability to maintain SA or manage the operation.
A case in point came from CC5, who explained that management and SA were the big ticket
items for him (CC5: 190). However, he outlined that they may be difficult to assess, and
deficiencies in these areas may show up in poor flying skill.
109

Chapter Five
R:

And you also suggested that sometimes, you said here that technical skills may
in fact be one of the first indicators that

CC5: Correct.
R:

management skills and situational awareness aren't coming up. Is that

what?

CC5: Yeah, what I'm saying is when you bring that all together it sometimes won't
necessarily show as below standard in management, it might be below
standard in technical skill (R: yep) but it's in fact because of, um, the inability
to be able to do both I guess. (CC5: 191-194)

5.4.4 Aviation knowledge

The MAPP shows that three constituents - SA, decision making and flying - were essential
during the assessment of pilots performance during promotion to airline captain. Additionally,
it became clear that the check captains were expecting candidates to have a sound level of
aviation-related knowledge. This knowledge included facts about aircraft operating systems, air
traffic control requirements, and air traffic control boundaries. What was important was how
candidates used aviation knowledge to develop SA, so that the aircraft was maintained within
required parameters. Initial verbal assessment of a candidates knowledge commenced prior to
the practical component of the assessment: Part of the briefing was that you do a little bit of,
um, knowledge testing at the same time (CC5: 45). In this case, CC5 was outlining that he was
ascertaining the level of knowledge the candidate possessed.

110

Chapter Five
Another case in which a candidates level of aviation knowledge affected their performance
was discussed by CC1:
he slipped into the holding pattern at 300 knots instead of the 230 knot maximum or
whatever it was at (R: yep) that point in time, I cant remember now, but he certainly
wasnt cognisant of the speed limitations in the holding patterns. (CC1: 213)
This misunderstanding or lack of awareness about particular operating requirements led to the
development of a flawed SA model by the candidate. The same candidate displayed a lack of
aviation knowledge during departure procedures:
he was briefing for the new runway, which he did quite capably (R: hm hm), but
totally missed the point that he had the wrong power setting in the computer. It defaults
to full thrust setting, so he was erring on the right side, but the fact is he didnt pick up
on how the bloody thing works. (CC1: 190)
In this example, the candidate was unaware, or failed to demonstrate an awareness, of the
operating functionality of the flight management computer. It was clear to CC1 that this
candidate was lacking in aviation knowledge. This was affecting his SA, his decisions, and the
way the aircraft was being flown. The candidate was assessed to be below standard.

A vital area of aviation knowledge was an understanding of, and the ability to, operate aircraft
within company standard operating procedures. The check captains saw these descriptive
procedures as a fundamental element of aviation knowledge for candidates to possess.
Comments like look, that was outside either company procedures or accepted performance
criteria (CC2: 37) demonstrated a concern. In this case, the candidate was not operating the
aircraft within the bounds of standard operating procedures. Another candidate did not carry
out specific procedures after take off: The other thing that stuck in my mind, uh, on one of the
take-offs; there was something unusual about his take-off scan. It was, uh, done earlier or later

111

Chapter Five
than general people do it (CC4: 174). In both cases, each candidate was not successful in being
promoted during these assessments.

An issue during the analysis was determining if aviation knowledge was an essential or
enabling constituent. For instance, when asked about aviation knowledge, CC4 did conduct
initial testing through questioning, though there was no real emphasis on stating if a level of
aviation knowledge was expected.
R:

And when you question, do you stop at a few or do you ...?

CC4: It depends on the response, you know, if its whatever it is, if the responses are
correct or very close to correct, youll quickly get, um, the feeling that this guy
has been doing his reading, hes being doing his homework and understands
the system that youre enquiring. (R: Yep, okay) But if he, if his answers are
wide of the mark, well then you have to ask a bit, delve a bit deeper to see
where the problem is. Whether its his inability to grasp the topic or he just
hasnt been doing it, hasnt been doing his homework. (CC4: 29 - 30)
One of the main issues that placed aviation knowledge as an enabling, rather than essential,
constituent was that the check captains were focused on how the candidates used the
knowledge to inform other areas, such as SA, decision making and flying. Additionally, there
were no assessments that were discussed with any of the check captains that were marginal due
only to aviation knowledge.

5.4.4.1 Linking aviation knowledge with SA, decision making and communication
The ability to develop SA and make sound decisions has been discussed as essential. However,
the check captains acknowledged that aviation knowledge assisted these essential areas. As
would be expected, there are numerous limitations required when flying aircraft around busy
airports. Some of these limitations may be related to avoiding terrain or controlled airspace. In
112

Chapter Five
the following instance, a candidate was flying an aircraft legally, though in close proximity to
uncontrolled airspace and high terrain. CC1 was concerned about his ability to draw a clear
link between the candidates aviation knowledge about control area steps (which is related to
air traffic control boundary), the decision the candidate made, and the candidates SA:
he had no idea that he was sitting on the control area step, because if he did have an
idea, then the least he could have done was turn around to his first officer and say I
accept the fact that Im sitting on the area step, but thats what they told me, (R: hm
hm) and its legal (R: yep). I would have said that, I would have said yes, Im aware
of the fact that were on the bottom of the step, you know (R: yep). This is
communication, this guy is not communicating (R: ok). He didnt say anything to me
whatsoever. (CC1: 221-223)
CC1 in this situation was concerned on a number of grounds, including an unease that the
candidate was developing SA on incorrect aviation knowledge, or the SA that the candidate
had was correct, and the decision to operate the aircraft close to terrain was too risky.
However, in this example, the reasoning behind the decision to operate the aircraft in a
particular way was not clear to CC1. If the decision and reasoning were relayed to the FO, they
would have been able to assist in improving the SA of the candidate, or relay their concern
regarding the risk associated with the decision. In this case, the candidate was below standard.

5.4.5 Management of crew activities

Management of flight crew activities was considered an important skill during promotion to
airline captain. However, the level of importance placed on management differed between the
check captains, with some not finding it an entirely critical component. Despite these
differences, like aviation knowledge, if management was taken out of the structure for
assessing performance, the structure did not hold. For this reason, management was
conceptualised as an enabling constituent that assisted candidates in maintaining SA, decision
113

Chapter Five
making, and flying. One example demonstrates that management can be considered as
enabling. During an assessment in an aircraft, a candidate carried out a poor landing. Although
CC2 was surprised with the heavy landing, he adopted a practical approach, stating thats fine,
these things happen (CC2: 247). The situation was initially handled well, with the candidate
ignoring the landing, continuing to taxi the aircraft to the terminal and carry out all the
appropriate check lists. The candidate made decisions regarding whether the landing was
categorised as heavy, and if it required an engineer to do a heavy landing inspection. However,
after departure to the next aerodrome, the cabin crew, utilising the aircraft cabin
communication system (service interphone) began to convince each other that the landing was
hard enough to have caused injury.
CC2: ...and then on the way to, so we went, we took off and um on the way to
[capital city airport] I happened to be listening to the service interphone, so I
heard all the cabin crew talking to themselves. One down the back would sort
of say I got a stiff neck, Yeah I got a sore back. (CC2: 251)
The combination of the landing, and the evolving situation with the cabin crew, was almost
beyond the coping skills of the candidate. As the candidate was finding the situation difficult,
CC2 decided to assist in the operation.
CC2: So I said to the candidate, I said Listen, you know, I missed the conversation,
but this is whats going on, you know, what are you going to do about it? (R:
yep) And again, you know, not particularly well handled. Difficult, you know,
for a young guy that, you know, without a lot of the airline background. (R:
mm) As we were checking people in line probably a little bit quick for some
people, (R: yep) and so he gave me his thoughts. A bit like pulling teeth trying
to get them out so, (R: yep) and the end result, so we landed at [capital city
airport]. And I must admit, to my annoyance, I took over [the management of
the aircraft], because it just wasnt going to happen if I didnt. (CC2: 251)
114

Chapter Five
Even though the management of the situation was not handled well, CC2 considered other
parts of the operation made up for this deficiency. Later in the interview, the following
question was asked about this candidate:

R:

So that guy back that you checked, how was his SA? Was his situational
awareness reasonably good, the guy who planted it?

CC2: It was satisfactory. Yeah. But it wasnt outstanding. (CC2: 273)

The above candidate was passed, although his management skills did not meet the expected
standard of CC2. However, his SA was assessed as satisfactory. This indicates that SA
remained an essential constituent, though management was enabling.

5.4.5.1 Prioritising crew activities and operational tasks

There was a view by the check captains that prioritisation was a fundamental component of
managing crew and other personnel. Candidates were required to prioritise operational tasks so
that aircraft was maintained within specified parameters, and SA was maintained. To be a good
commander they need to understand that really all they need to do is be a good coordinator
(CC4: 211). In the next example, a candidate is seen as not prioritising crew activities. Instead,
the candidate was more focused on contacting a company engineer to fix an unserviceable
piece of navigation equipment. However, the first priority should have been the landing of the
aircraft.
I keep notepads and it was, you know, not quite what I wrote, but it was Land the
sucker. You know, weve started to get distracted about, (R: okay) you know, so there
you are, 30 miles out of [regional airport], rocketing in to do an instrument approach
and were trying to talk to company operations on ground for starters. (R: yep) It was
not prioritising the important things (R: sure) very well. (R: mm) So from that point it

115

Chapter Five
was starting to get untidy. But then off the missed approach, when he wound in 300
knots, I thought Oh no! This will be interesting! (CC2: 132)
This candidate was required to be reassessed by CC2.

Another case of poor prioritisation was a candidate coordinating crew activities by delegating
the task of taxiing to the FO whilst he investigated a faulty engine instrument. In this situation,
the candidate, a previous engineer, had identified that the prime engine power indicator (N1)
was indicating a higher than normal RPM during idle. However, CC4 believed the candidate
should have delayed dealing with the problem until they arrived safely at the terminal gate.
CC4: we were taxiing in and, um, the idle RPM on one engine was a bit higher
than the other, (R: okay, yep), the N1. And he actually got a book out and
looked up what it should be and you know, it was a bit high. He was quite
right, it was. But I thought with flight management wise, when youre taxiing
in, its not a good time to be getting books out and checking the N1.
R:

So he actually got the first officer to taxi or was he taxiing in?

CC4: Yeah, no he got the first officer to taxi and he got (R: yep) the book out and
looked it up (R: yep). And I didnt really, you know, he could have just written
down what it was or made a mental note of what the actual reading was and
looked it up in five minutes when I put the park brake on, that would have been
more sensible. (CC4: 171-173)
This example showed a candidate displaying poor prioritisation of crew tasks, and was judged
to be below standard by CC4.

5.4.5.2 Linking management to SA

The next case in point demonstrates management as fundamental to maintaining the
candidates SA. This case is representative of management - and in particular, the ability to
116

Chapter Five
prioritise crew activities and operational tasks - being intertwined with SA. In this example, the
candidate, whilst being assessed on how he dealt with emergencies, was delegating the flying
of the aircraft to the FO (in the example below, the candidate would say your aeroplane). This
allowed the candidate to directly deal with the emergency by reading the emergency checklist
(QRH which refers to the quick reference handbook) himself. However, CC3 was concerned
that this management approach placed the candidate in a position where they were unaware of
the aircrafts position.
CC3: as there was a problem, he would get the, say to the guy, your aeroplane,
and he would get the QRH out and he'd do all the reading. (R: Yeah). And then
look up and say, Oh, where are we now, what is going on? Now the Captain
should never be out of the loop, and the Captain cant be out of the loop. The
FO can be out of the loop if so need, but the Captain must always know where
the aeroplane is at (R: yep) and what is happening at all times. And for him to
be doing all the checklist reading and delving and trouble shooting, it was, the
priorities were all wrong.
R:

So for you, he had lost his ...

CC3: SA. (CC3: 257-261)

The action of the candidate, by reading of the QRH, was such that he managed the operation in
a way that allowed him to reduce his SA. This was unacceptable to CC3, and clearly brings
together the idea that candidates are required to manage the operation so that they are able to
maintain SA. This candidate was seen as below standard due to loss of SA.

5.4.6 Communication amongst crew

Already, the thread of communication has worked its way through most constituents, whilst
assessing a candidate for promotion to airline captain. Communication has been imbedded into
SA, decision making, and management. However, like aviation knowledge and management of
117

Chapter Five
crew activities, communication was identified as an enabling skill. For example, during an
assessment that had been conducted by CC1, a candidate was seen to have poor
communication skills. However, the candidate was capable enough to maintain essential
constituents throughout assessment.
R:

Ill throw one at you then. If hes a poor communicator but has good
situational awareness, and a little bit hypothetical, its all very difficult, but
would you be happy, do you think to pass someone like that? Would you

CC1: Yes, as he doesnt fall over in other areas. One comes to mind straight away.
He gives the impression of being the most dithering young fellow, and I did his
check. I had long, long, and hard words with him, and Ive done line checks
with him as well. I did his final check and I also did line checks with him
afterwards, and every time was the same. I was a bit disappointed because he
obviously didnt take on what I said or wasnt capable of changing, but his
PAs were awful (R: yep). Ums and ahs and buts and, and his communication
with his first officer was, was almost stuttering and stammering. (R: hm hm). It
was so unconfident. But he knew what he was doing, he flew the aeroplane
well and he knew exactly where things were and hes still a very capable
captain. (CC1: 359-361)
The example above shows that even though communication is an important component of the
MAPP, it was only seen as an enabling constituent.

5.4.6.1 Linking communication with other constituents

As outlined, communication has already been linked with other constituents. For instance, as
an enabling constituent, its importance in coordinating crew activities was certainly
emphasised by three of the check captains. Additionally, it was used by candidates to be able to
share their SA, and intended course of actions with other crew during decision making. For
118

Chapter Five
example, CC5 held a senior managerial position within his airline. He described an interview
with a candidate that was conducted due to a recent failure during a final simulator assessment.
The interview centred on how the candidate had conveyed to the FO his decision regarding
how the aircraft was to conduct the approach and landing into a capital city airport. As the
weather was marginal, two options were available - a full instrument approach, or an
instrument approach with a bad weather circuit. In this assessment, the aircraft was tracking to
a navigation aid called a non-directional beacon (NDB) to enable the aircraft to land in poor
weather. It was clear that there was a mismatch between the expectations of the candidate, and
the expectations of the FO. This mismatch was due to poor communication on the part of the
candidate.
the captain [candidate] elected to ask him to do an NDB onto the runway instead of
doing a legal bad weather circuit to stay up out of the soup. (R: yep) And, ah, even
though it was fine, (R: yep) and then that wasn't communicated clearly to the FO, he
thought they were just tracking around the NDB to stay high and not doing an
instrument approach. (R: ah okay) So the whole thing fell apart due to lack of, you
know, it appears, we haven't had the meeting yet, but it appears due to poor
communication between the captain (R: yep) and his first officer as to what they were
really trying to achieve. (CC5: 131)

There was also an emphasis on candidates acting in ways that facilitate the effective flow of
information. This information flow assists all crew in maintaining and improving SA, as well
as developing a shared understanding of decisions and intended plans. The flow of information
required the candidate to effectively transmit to, and receive information from, the crew.
Candidates behaviour can either encourage or inhibit this flow of information.

119

Chapter Five
During an assessment, a medical problem with a passenger in the cabin required the candidate
to converse with cabin crew, on a radio interphone. In this example, CC5 is outlining how the
candidate is required to communicate with cabin crew, so as to obtain as much information
from the cabin as possible, so that a decision to either continue to the destination or divert back
to the departure aerodrome could be made.
CC5: this particular exercise was more around communication. It's actually
finding out what's going on in the cabin and determining, making decisions,
some decision making obviously as to what are we going to do, are you going
to continue on to [capital city airport] or make it back? And then sourcing as
much information as you can to make that decision. (CC5: 212)

Communication is an enabling constituent for candidates, as it assists in maintaining and

improving SA at an individual and crew level. It is also foundational to improving decisions,
and relays the intended requirements of the decisions made. Additionally, candidates are
required to be able to promote effective communication among crew.

5.5

Summary of the findings

The findings have shown that check captains are working with specific constituents, which
they use in assessing whether candidates have reached a minimum standard for promotion to
airline captain. Specifically, the findings have demonstrated that SA, decision making and
flying skills, are essential constituents that are used. Additionally, the findings demonstrate that
candidates employ enabling constituents, like aviation knowledge, management of crew
activities, and communication, to increase the likelihood of maintaining essential constituents
at or above a minimum standard.

120

Chapter Six

Chapter Six
Discussion and Conclusion
6.1

Introduction

This study set out to investigate the criteria used to assess the performance of pilots during
their promotion to airline captain. As outlined in Chapter Two, the regulatory authority and
airlines have detailed measures for both technical knowledge and skills, which airline pilots are
required to demonstrate during assessment. Interestingly, this study has shown that, although
technical skills are important, some nontechnical skills are equally necessary to pilot
performance. This phenomenological study found that, during promotion to airline captain,
NTS such as SA and decision making are just as essential as flying skills. Supporting these
essential areas are the enabling skills of aviation knowledge, management of crew activities,
and communication amongst crew. This chapter now examines the contributions the study will
make to the aviation industry, including a better understanding of the criteria currently used to
assess pilots performance, and the relationship between the criteria. Additionally, this new
awareness about pilot assessment was instrumental to the development of a singular model
(MAPP), which brought together both technical skills and NTS. The potential wider
application of the MAPP as a means for assessing skilled performance in other vocations is
also considered as a secondary benefit to this research. Finally, there is discussion on using
phenomenology, as an effective methodology for research into some aviation problems.

There will be a discussion of several implications for aviation training and assessment.
Recommendations will be made throughout the chapter, based on the implications of the

121

Chapter Six
research outcomes. The thesis will conclude with a reflection on the initial problem identified,
and the some of the solutions that the study has generated.

6.2

The problem and how it was addressed

Advances in aviation safety have, in part, been attributed to the strong emphasis on assessment
of pilots technical skills and knowledge (Johnston et al., 2000). Although this approach to
assessment was identified as important, the aviation industry now acknowledges the
significance of NTS in pilots performance (Helmreich & Foushee, 1993; Maurino & Murray,
2010; Thomas, 2003, 2004). Recently, assessment metrics have been introduced into some
airlines to assist in measuring pilots NTS, with the most recognised assessment tool being
NOTECHS (Flin & Martin, 2001; Flin et al., 2003b, 2005; Flin et al., 2009; Wilson et al.,
2010). Some of the measures to assess NTS are detailed; these have been seen as complex
(Johnston et al., 2000), and sometimes overly prescriptive (CASA, 2009e). As a result, airlines
have found measures to assess NTS difficult to use in practice.

As noted in Chapter Two, a number of reasons have contributed to the difficulties that airlines
were experiencing in implementing NTS assessment, including the aviation industrys
continuation to train and assess technical skills and NTS separately. For instance, Appendix 1
and 2 illustrate the metrics developed to assess pilots performance using two separate methods
for technical skills and NTS. This was argued in Chapter Two to be problematic, with the
aviation industry placing an emphasis on assessing technical skills, rather than determining the
particular skills area that were important. Additionally, it was argued in Section 2.6 that the
fragmentation and complexity of NTS was a reason why its assessment might be encountering
difficulty. For example, some of the NTS criteria that were developed, such as teamwork and
cooperation, were seen to have areas of overlap and duplication (see Section 2.4.3 and 2.4.5).
122

Chapter Six
Furthermore, those researching other NTS areas, such as decision making, developed theories
that were not reflective of practice (see Section 2.4.2). As such, Chapter Two concluded that
one way forward would be to select a research approach that overcame fragmentation by
integrating, and attempting to capture, the complexities associated with pilots performance.

A research methodology was sought that captured individual pilots performance at or around
a minimum standard during promotion to airline captain. It was acknowledged in Section 3.2
that great advancements in areas like medicine and psychology over the last century were
accredited to the scientific practices that viewed substantial knowledge claims that were
authenticated with empirical data or studies based on a positivist view (Crotty, 1998; Schn,
1983). However, it was identified that positivism was not as effective at identifying a range of
phenomena that are not able to be described in positivistic terms, such as instability,
uncertainty, value, or conflict (Schn, 1983). In support of this, Husserl established a view that
a subjects consciousness was not separate from the world, but a means to experience the
world (Moran, 2000). For Husserl, nothing could be even spoken about or witnessed if it did
not come through someones consciousness (Giorgi & Giorgi, 2008, p. 26). As such,
neglecting the consciousness of humans was seen as problematic. These arguments aligned
with the problem the author identified in research methodologies in the field of NTS, where
continual fragmentation was seen as not solving the problem, but exacerbating it.

Seeking alternatives to positivism was seen as an important step forward into gaining valuable
insight into pilots performance. As such, social constructionism was viewed as providing a
means of grounding the epistemological claims for the current study. Moreover,
phenomenology was chosen for a number of reasons, including that it allows the researcher to
capture a holistic view of the issue, or to determine the psychological meaning (Camic,
Rhodes, & Yardley, 2003, p. 8). This approach enables one to see knowledge as interrelational
123

Chapter Six
and structural, interwoven in networks, which allows the researcher to investigate
phenomenon, and more importantly, relationships between phenomenon (Kvale, 1999, p. 101).
Phenomenology also enables lived experience to be explored in-depth (Giorgi & Giorgi, 2008).
Finally, due to limitations of positivistic research (Sandberg, 2005), alternative approaches to
investigating performance at work were evolving (e.g., DallAlba, 2002; Sandberg, 2000,
2005; Webster-Wright, 2006). These problems gave rise to the current study.

6.3

Contributions of this study

The current study investigated the experiences of five highly experienced check captains in
order to develop a better awareness into the assessment of pilots performance during
promotion to airline captain. An important outcome of this study is the contribution it will
make to an improved understanding of the current technical skills and NTS used to assess
pilots performance, and the relationship of these two areas. As a consequence, this
understanding has lead to the model for assessing pilots performance (MAPP). The findings
also, potentially, have wide-ranging implications on views about skilled performance in other
vocations. Finally, the research has contributed to a broader view of research approaches in
aviation.

6.3.1 Improved understanding about technical skills and NTS in pilot assessment
In the early stages of the current study, it was identified that little research had investigated the
contemporary practice of check captains conducting assessment of pilots performance. In
particular, no research was found that focused on the performance requirements of pilots
undergoing promotion to airline captain. Accordingly, an important contribution that the
current study makes is a better understanding of the required technical skills and NTS in pilots
performance. The study demonstrates that, in current practice, check captains assess both
124

Chapter Six
technical skills and NTS; i.e. flying skills, SA, decision making, aviation knowledge,
management and communication. Therefore, the current study has identified the key criteria
used to assess pilots performance.

In addition, the findings unveil the relationship between these assessment criteria, allowing for
an improved understanding of the required performance for pilots. As outlined above, it was
identified that the check captains use six different criteria during assessment. What emerges
from the findings, however, is that all key criteria are not considered equal. The ability to fly
the aircraft within regulatory parameters, SA and decision making were given a greater
emphasis than aviation knowledge, management of crew activities, and communication
amongst crew. However, these findings contradict present theories on how NTS are viewed.
For example, Figure 15 proposes how current NTS are assessed. As can be seen, if any of the
current criteria (see Appendix 2 for examples of elements and criteria) were identified to be
below standard, this would lead to a fail grade. However, the findings in the current study did
not support this proposal. For instance, the research revealed that candidates could pass an
assessment if they were able to maintain the three essential skills of flying, SA and decision
making. This was even if they were not performing well in other NTS areas. The NOTECHS
assessment model of assigning an overall fail grade to candidates, who fall short on any one of
the criteria, does not take into account the fact that certain NTS are more critical than others.
This new insight into how NTS are viewed is a major contribution to understanding how pilots
performance is assessed.

125

Chapter Six

Figure 15 The NOTECHS Hierarchy of Levels

Taken from O'Connor et al. (2002, p. 267)

In light of the above, the current study identified a hierarchy of specific criteria currently used
to assess pilots performance. However, the study was unable to fully explain, in detail, the
stand alone measures for each criterion. For example, what would constitute an acceptable
level of SA? As such, further research is needed to investigate how each individual criterion
influence pilots overall performance. As outlined, Sadler (2005) has concerns for the
interchangeability of the terms criteria and standards. He argues that simply having detailed
criteria does not define a standard. The current study has plainly shown that assessing pilots
performance, using all criteria equally, is problematic. However, the current study is only the
first step in better understanding pilot standards.

Recommendations
These contributions lead to three recommendations:
Further research to confirm the key criteria used to assess pilots performance;
Further research to confirm the relationship between key criteria; and
Further research to identify appropriate standards for each criterion.
126

Chapter Six
6.3.2 Development of a model for assessing pilots performance (MAPP)
Based on the enhanced understanding of the criteria, and the relationship of the criteria used to
assess pilots performance, a model was developed for assessing pilots performance (MAPP).
During the current study, the MAPP was presented to the wider aviation community for
critique, and also trialed in an international airline as a method to assess pilots performance.
As can be seen from the responses and evaluation outlined in Section 4.6.1.4, the MAPP has
been shown to be a viable approach to assist in the assessment of pilots performance. Its
strengths lie in its simplicity to draw together the criteria required during assessment of pilots
performance. In particular, the MAPP allows check captains to assess pilots use of criteria that
are reflective of current practice. It is argued that, prior to this point, assessments that were
against the better judgments of the check captain could have been made using NOTECHS.

It must be stressed that although the findings and, in particular, the MAPP have been trialed, it
is important that further research and industry consultation be conducted. The focus of such
activities is to develop a clear view of what is ideal practice, rather than what is outlined, which
is current practice.

Recommendations
Further research into the use of the MAPP for assessing pilots performance; and
Further research and industry consultation into whether current practice is ideal
practice.

6.3.3 Potential for a wider application of the MAPP to assess performance

The focus of the research was to determine which criteria, and the relationship between the
criteria, are used to assess pilots performance in undergoing promotion to airline captain.
However, the MAPP may be relevant to groups beyond this study. For instance, there is some
127

Chapter Six
support for the models applicability to other pilots, such as first and second officers. During a
training program that was used by the researcher to test the model (see Section 4.6.1.4), eight
pilots were assessing video scenarios involving captains, first officers and second officers. The
MAPP was used effectively for all three job positions, to assist the assessors to assess the
scenarios. As such, the model appeared to be transferable across positions in the flight deck. In
addition to the model being transferable to other pilot roles, during a presentation of the
research findings in New Zealand (see Section 4.6.1.4); an experienced air-traffic-control
instructor noted that the model would be applicable within his training program.

The MAPP assists an assessor to place an emphasis on specific criteria used during assessment.
In particular, it allows technical skills and NTS to be seen collectively, and as a hierarchy. The
potential comes with many industries such as engineering, medicine, police, fire and rescue,
and the armed services now identifying NTS as being important within their fields (Flin et al.,
2003a; Flin & Maran, 2004; Flin et al., 2009; Helmreich, 2000). As a result, the research
findings potentially have some relevance to other professions, which have their own unique
technical knowledge and skills, but are also now discovering the importance of the generic
NTS concepts. This potential for a wider application is supported by other researchers, who
draw on models and theories of vocational performance, development, and practice, to better
understand how individuals perform within their particular workplace (Billett, 2001; Billett &
Stevenson, 2000; DallAlba, 2002; DallAlba & Sandberg, 2006; Dewey, 1938; Dreyfus,
Dreyfus, & Athanasiou, 1986; Ericsson & Lehmann, 1996; Evans, 1994; Stevenson, 1994,
2003). The current study may contribute to understanding the multifaceted nature of skills,
within complex work environments, by assisting in the understanding of the key requirements
for performing tasks, as well as the development of assessment methods to judge performance,
in relation to these tasks.

128

Chapter Six
Recommendations
Given the potential for wider application of the MAPP in assessing performance, the following
recommendations are made:
Further research be conducted into the key criteria used to assess performance in a
variety of professions, as well as standards used;
Further research be conducted into the relationship between key criteria in a variety of
professions; and
Research and consultation with a variety of industries to determine if current practice is
ideal practice.

6.3.4 Methodological contribution to aviation research

The use of phenomenology as a research methodology is unusual in aviation research. When
the researcher searched both qualitative and phenomenological methodologies in a well
regarded international journal - The International Journal of Aviation Psychology - of the
almost 450 articles in print between 1991 and 2010, only one had used a phenomenological
methodology. Although aviation colleagues had expressed some concern to the researcher
about using such an atypical approach, the problem of fragmentation that had been identified
early in this thesis was suited to a phenomenological methodology. Rather than falling into line
with methodologies that are typical of those used in aviation, a methodology that appeared
more suitable was selected. A major contribution that this study makes is to demonstrate the
viability of phenomenology as a research methodology in aviation.

Recommendation
During research into issues related to aviation, there is a need to consider a broad range
of research methodologies that best suit the problem.

129

Chapter Six

6.4

Implications of this research

The results of this study have implications for significant changes to several areas within
aviation training and assessment: the future direction of pilot assessment; improved validity
and reliability in pilot assessment; increased assistance in the development of assessment
scenarios; reduction in the likelihood of conflict during assessment; the improvement of
training for check captains; improved training approaches for pilots; and finally, modifications
to current policy in aviation.

6.4.1 Future direction of pilot assessment

Research shows that, in some professions, the assessment of an individuals performance lacks
a level of rigor (Sadler, 1987, 1989, 1998, 2005; Salas, Bowers, & Prince, 1998). Aviation
appears to be no exception, with difficulty assessing individuals (Brannick et al., 2002) and
inconsistency existing between, and within, airlines (Goldsmith & Johnson, 2002). For
improvements to occur with these acknowledged inconsistencies, the findings of this study
suggest changes in the future direction of pilot assessment in two ways, as outlined below.

To begin with, this study demonstrates the need for a new model of pilot assessment in the
aviation industry, which acknowledges a hierarchy within the current criteria used to assess
pilots performance. If such a model were adopted and, in particular, the MAPP, could this
assist in developing improved assessment instruments? As can be seen in Figure 16, the
current approach to pilot assessment views technical skills and NTS as separate. However, it is
suggested that improvements can be gained by uniting all criteria into a single system, based
on the MAPP. This would produce a more relevant and effective single instrument to assess
pilots performance. As outlined in Chapter Two, this is reflected by Sadler (2009), who

130

Chapter Six
suggests that to develop standards requires a sound theoretical understandings of performance
(refer to Section 2.7).

Technical skills
criteria

Current practice sees

criteria as separate

Nontechnical skills
criteria

MAPP acts as filter to

better determine standards

Situational Awareness

Essential

Enabling

Aircraft flown
within tolerance

Knowledge

Decisions
considerate of risk

Management

Communication

New pilot standards based on filtering old criteria through the

MAPP, to develop new single system for measuring pilots
performance.

Figure 16 Illustration of how MAPP can develop improved assessment

measures for pilots

Second, if future assessment models resemble the new single system for measuring pilots
performance (see Figure 16), there are still questions relating to what importance should be
placed on each of the criteria. The current study established three areas of pilot performance as
essential: flying skills, SA and decision making. However, it was not within the scope of this
study to investigate whether only the three essential skills should be assessed, or whether all of
the criteria are to be met, with any subsequent poor performance in the essential skills being
131

Chapter Six
seen as a failure. As outlined in Section 6.3.1, further research and industry consultation is
necessary to develop more robust criteria for each skills area if a new single system for
assessing pilots performance is to be integrated into practice.

Recommendations
The recommendations for this section are aligned with those outlined in Section 6.3.1 and
Section 6.3.2:
Further research to confirm the key criteria used to assess pilots performance;
Further research to confirm the relationship between key criteria;
Further research to identify appropriate standards for each criterion;
Further research into the use of the MAPP for assessing pilots performance; and
Further research and industry consultation into whether current practice is best practice.

6.4.2 Improved validity and reliability in pilot assessment

It has been stated that pilots performance is judged more frequently than any other profession
(Goldsmith & Johnson, 2002); however, a level of concern in relation to the consistencies of
assessment has been expressed (Brannick et al., 2002; Holt et al., 2002). This is why it is
important to discuss the implications that the research findings have on validity and reliability
in pilot assessment. For instance, the appropriateness and adequacy of assessment results refers
to the assessments validity. High validity implies that the assessment measures what it claims
to measure; low validity implies that the measure does not assess what it was designed to
assess (Linn & Gronlund, 1995). Four major considerations in the area of validity are content,
test-criterion relationship, construct validity, and consequence validity. If an assessment can
accurately verify, by a sampling of tasks, the ability of an individual or team to perform a given
task, the assessment is said to have content validity (Linn & Gronlund, 1995). This is not to be

132

Chapter Six
confused with test-criterion relationship validity, which is asking whether, if pilots are assessed
on a particular skill today, will they perform to the same level in two months time (Linn &
Gronlund, 1995)? Construct validity refers to the extent to which the measure assesses special
characteristics that may assist the assessor in predicting future behaviour (Beaubien, Baker, &
Salvaggio, 2004; Linn & Gronlund, 1995). For example, good decision making during
assessment, may be able to predict an individuals ability to deal with a real life emergency.
Consequence validity is identifying the positive and negative effects of the assessment (Linn &
Gronlund, 1995). For example, the stress that is related to continual assessment of pilots.

While it was not within the scope of the current study to investigate the test-criterion and
consequence validity of the assessments in pilots performance, the study did identify a
limitation in content and construct validity. When assessing an individual pilots ability to fly
an aircraft, by use of the current regulatory parameters (technical skills), it is assumed that, if
the pilot passes, they are indeed able to fly the aircraft in all expected situations. The current
study has, demonstrated that, although NTS are also important, they are not currently contained
within the existing measurement instruments, hence, content validity is placed into question.
Moreover, even if a NTS assessment measures, such as NOTECHS, is integrated into the
present assessment process, the results of the current study imply that not all criteria used to
assess pilots performance are equal (see Section 6.3.1). As such, the research identified an
inconsistency in construct validity. This claim has been supported by recent studies that have
found that construct validity in pilot assessment requires substantial improvement (Beaubien et
al., 2004). These improvements include better design of assessment scenarios, improved
assessment forms, and instructor training, all of which have been, or will be, discussed in this
chapter. Clearly, if improved metrics are used, such as those detailed in Section 6.4.1,
improving assessment scenarios and check captain training (Section 6.4.4 and 6.4.5 below),
there could be improvements in content and construct validity during the assessment of pilots
133

Chapter Six
performance. Whilst it has been shown that the study has implications for specific areas of
validity, it is also suggested that the study may enhance reliability during assessment.

Reliability is concerned with the measurement of performance over different assessment

sessions, usually with different assessors, on individuals performing the same tasks, and is
focused on the consistency of assessment results (Linn & Gronlund, 1995, p. 48). For
example, if a pilot was being assessed by a check captain, and the entire session was videoed,
high reliability would occur if another check captain, watching the same session, gave the same
assessment scores.

Figure 16 provides an improved method for describing validity and reliability. The capacity to
hit the target or average around the target is validity. The ability to be consistent is related to
reliability, with close proximity of the groupings relating to reliable assessment, and unreliable
results occurring with wider groupings.

Figure 17 Illustration of validity and reliability in assessment

Modified from Linn & Gronlund (1995, p. 48)

Using the above targets as an illustration, the following will explore the implications that the
current study has for the present regulatory system in aviation. There could be a perception that
134

Chapter Six
the current regulatory system of assessment produces valid and reliable results (Target 1).
However, while the current metrics that are in place to measure pilots performance may assist
in developing reliable outcomes, validity has been questioned by the current study, with check
captains identifying that NTS are also required to be assessed. As such, the current assessment
process is represented by Target 3 in Figure 16. That is, it appears to be reliable, but is less
valid. Moreover, if NTS are introduced in their present form, without appropriate standards,
there could be a possibility of becoming more valid, with reliability in NTS assessment being
identified as difficult to achieve (Brannick et al., 2002; Holt et al., 2002). Target 2, with check
captains applying their interpretation on NTS, would best represent this scenario. Reliable and
repeatable assessment of pilots performance is unlikely if assessors do not understand and
agree on standards (Baker & Dismukes, 2002). The implication of the research is that, to make
the assessment process more valid and reliable (towards Target 1), there needs to be an
integration of NTS into the current assessment measures, with appropriate standards developed
(such as those outlined in Section 6.4.1).

6.4.3 Reducing the likelihood of conflict during assessment

The above section outlined the fact that the current study may have a positive influence on
validity and reliability in assessment. This section continues with the implications that
improved assessment processes may have during assessment. The phenomenological approach
enables in depth research (Giorgi & Giorgi, 2008), and as a result, other findings may arise
during the investigation. For instance, this research did not adopt any preconceptions regarding
the identification of conflict as an issue during assessment, though it did arise during
interviews. To explain, reliability or consistency between assessors has been argued to be
important (Baker & Dismukes, 1999) and has been discussed above (Section 6.4.2). The
current study identified that check captains were assessing both technical skills and NTS.
However, the NTS were not being assessed against clearly identified criteria, nor standards.
135

Chapter Six
This may increase the likelihood of conflict (Cooksey, 1996). Interestingly, the potential for
judgments that are not based on clear criteria and standards, increasing the possibility of
conflict was supported in this study. This arose with comments like that shown in the following
example, in which the candidate was having difficulty understanding why he had failed.
CC2: And he had some points, I mean hes shown to me a couple of times,
Ive come across him in a car park, and and hes come up and
said I want to talk.
R:

Okay.

CC2: And it goes nowhere of course. You have actually got to be very
careful getting drawn into those sort of (R: sure) because youll say
something youll regret. (CC2: 95-97)

In another example, the candidate was unhappy with the check captains judgments about
requiring more training.
CC4: Yep. Uh, so I said that he needed to do a bit more training on the
aeroplane in another twenty sectors or something (R: mm) before Id
be happy to sign him out. (R: okay). But the instant that I told him
that I think he needed to do some more training so he understands
the system and knows what needle hes tracking on, the instant I told
him that, he went really quite angry. He stood up and he thumped the
table and went really red in the face and said I was only picking on
him and (R: uh huh) threw the book down and stomped out of the
room. (CC4: 162)

To move away from relying on intuitive judgments, and to reduce the likelihood of conflict
during assessment, a level of resilience needs to be built into an assessment process. Resilience
136

Chapter Six
would mean that individuals being assessed are given grades or scores that are reflective of
their performance. This point is aligned with the concept of grade integrity (Sadler, 2009)
outlined in Chapter Two (see Section 2.7). The implication from the current study is that
improved understanding of criteria and standards will allow for a more transparent grading
model (e.g. using the MAPP). Grades given will be more reflective of a thoroughly informed
theoretical understanding of performance and standards, and as such, may reduce the likelihood
of conflict during assessment.

6.4.4 Assisting in the development of assessment scenarios

During the assessment of pilots performance in a simulator, the designer develops either one
or multiple scenarios. These scenarios are commonly referred to as event-sets. For example, a
designer may have two events-sets during an assessment. The first may be a sixty minute
event-set with a normal departure, followed by a hydraulic malfunction on climb, necessitating
the crew to determine an appropriate cause of action. What may follow is a shorter fifteen
minute event-set, which has the aircraft already in the air, with the crew being required to deal
with a pressurisation problem. Typically, using longer event-sets necessitates assessors to
mentally collate the performance of pilots, and make a final judgment. As a consequence, it
was recognised that by utilising shorter event-sets gave greater consistency in assessment
(Seamster, Edens, & Holt, 1995). Nevertheless, it is acknowledged that developing event-sets
is time consuming, and that those that are used do not always measure NTS effectively (Baker
& Dismukes, 2002).

It appears that the current study may have an implication for the development of event-sets,
and hence assessment of pilots overall. This was discovered whilst trialing the MAPP during
assessment scenarios (refer to Section 4.6.1.4). It was identified that improved consistency
occurred between pilots in the class by encouraging them to focus on essential skills outlined in
137

Chapter Six
the MAPP, and reviewing how enabling skills could improve overall performance. As a
consequence, it might be argued that longer event-sets, which appear to be the preferred
method, could now be used, given that the MAPP may assist those involved in assessing pilots
performance to maintain an awareness of the required essential skills. As has been mentioned
previously, research and industry consultation must still occur to determine the MAPPs
effectiveness. However, it could be suggested that, if the MAPP is found to be a viable model,
then reducing time to develop event-sets, and hence the costs associated with this task, may be
an implication for the current study.

Recommendation
It is recommended that further research be conducted into how consistency in
assessment can be affected on differing length event-sets, by use of the MAPP.

6.4.5 Improved training in pilot assessment for check captains

Check captains ability to assess pilots performance can be developed and improved in a
variety of ways. Baker & Dismukes (1999) found that four methods were available for
improving assessor reliability. The first method, rater-error training, outlines common errors
made during assessment. For instance, some errors include halo effect, horn effect, central
tendency, leniency, severity, primacy, and recency (Baker & Dismukes, 1999; Woehr &
Huffcutt, 1994). Halo effect refers to receiving a high grade for good performance in just one
area. The horn effect, however, refers to receiving a low mark due to poor performance in just
one area. Leniency and severity occur when assessors have a tendency to provide excessively
high or excessively low marks, respectively. Central tendency, in contrast, is where assessors
have an inclination to group all marks in the middle. The primacy and recency effect has
assessors being affected by how students began (primacy) or finished (recency) their
assessment, as opposed to their overall performance (Baker & Dismukes, 1999).
138

Chapter Six
The second method to improve reliability in assessment was performance-dimension training,
which familiarises and trains assessors in the use of rating scales used during assessment
(Baker & Dismukes, 1999; Woehr & Huffcutt, 1994). For instance, check captains would be
familiarised with the rating scales particular to their airline, such as those in Appendix 1 and 2.
Another way to improve reliability was behavioural-observation training, which focuses on
examination of behaviour, rather than evaluation. It is suggested that there is a distinct
difference in observing performance, and the judgment of performance. For instance,
judgment processes include the categorization, integration and evaluation of information,
while observation processes include the detection, perception and recall or recognition of
specific behavioural events (Woehr & Huffcutt, 1994, p. 192). It is proposed by Woehr &
Huffcutt (1994) that halo and leniency errors are due to poor observation techniques. Finally,
the frame-of-reference training method, which emphasised the multidimensionality of
performance, defining performance dimensions, provides a sample of behavioural incidents
representing each dimension whilst also allowing practice and feedback (Woehr & Huffcutt,
1994, p. 192). This may include allowing check captains to grade an individual pilots
performance, or crews performance, by watching a video. The check captains then receive
feedback that will assist in improving their reliability during assessment.

The most effective way to improve reliability was found to be the frame-of-reference method
(Baker & Dismukes, 1999; Woehr & Huffcutt, 1994). However, it is proposed that the MAPP
developed in the current study could further improve understanding of performance, thus
making performance-dimension training more effective. To explain, Woehr & Huffcutt (1994)
outlined that performance-dimension training assists assessors to examine the pertinent criteria
during assessment, rather than making an overall decision about performance. They outline that
for training raters to recognize and use the appropriate dimensions on which ratings will be
required should lead to dimension relevant judgments as opposed to a more global judgment
139

Chapter Six
(1994, p. 191). The potential is that the MAPP developed in the current study better informs
the understanding of appropriate dimensions used during assessment of pilots performance.
As a result, training of check captains, via performance-dimension training, could be made
more effective.

Although performance-dimension training may be the more obvious area that the MAPP
improves, it could be argued that use of the MAPP may unintentionally improve more than one
area of assessment. As a consequence, an implication for the current study is that check captain
training programs, using all four training approaches, could improve the reliability of
assessment within an airline. This claim is supported by the training sessions that were
conducted with experienced check captains by the researcher (see Section 4.6.1.4). If check
captains have an improved understanding of the appropriate dimensions to assess pilots
performance, in the form of essential skills criteria, there is likely to be increased reliability
during assessment.

Recommendation
Further research is recommended on the use of the MAPP in check captain training.

6.4.5 Improved methods in pilot training

Although the focus of the current study was on assessment, there appears to be clear link to
how the findings may influence training of pilots. There are three implications to aviation
training identified here, including a focus of training outcomes, increased emphasis on training
that focuses on maintaining SA, and an increased importance placed on management principles
during training of airline captains.

140

Chapter Six
6.4.5.1 Focus on training outcomes
As the current study identified the criteria used to assess pilots performance, it would seem
logical that training programs place emphasis on pilots being made aware of, and trained to
maintain, essential skills. Pilots would be trained to focus and prioritise their activities on
maintaining essential skills during flight, whilst using enabling skills to assist in maintaining
essential skills. For example, a pilot may notice reduced levels of his or her SA during a
difficult event. If they are able to identify this situation, they will be able to implement
strategies such as slowing down the use of particular procedural checklists, referring to the
other pilots to assist in improving their SA, or shedding their workload.

6.4.5.2 Training that focuses on maintaining SA

In addition to training programs that develop pilots awareness of essential and enabling skills,
the findings of the current study suggest that, given the importance of SA, there should be
ways of teaching pilots to maintain and improve SA. Methods to improve SA for pilots differ,
with some approaches being based on the findings that have been identified in studies outlined
in Section 2.4.1. The research into SA errors identified that approximately 80% of accidents
were due to level 1, or perception, errors (Endsley & Robertson, 2000). The results of these
findings may be utilised to reduce the likelihood of perception errors (level 1) through
improved work practice and design. Although this study did not examine the differing levels of
SA, some of the SA errors that have been identified by the check captains appear to be
perception-related problems. The two examples below are from CC1 and CC3 describing
candidates making a level 1 error.
CC1: From memory the first time I picked up on it was when I noticed he
didnt have the raw data. (CC1:142)

141

Chapter Six
CC3: a guy had a fire in the rear galley (R: okay) and the girls were
giving him, ah, prompts with Just fired the fourth bottle into this
thing and it was still going no urgency and no forward planning.
(CC3: 49)

In addition to training programs that focus on perception errors, there may be other
implications for training from the current study. Within vocational education, research has
attempted to explain how adults learn, improve, excel in, and adapt to, differing work and
home environments. An approach suited to describing knowledge, understanding, and practice
is one proposed by Stevenson (1994). It illustrates how skills can be categorised into differing
areas. Stevensons work describes how differing levels of skills require different levels of
conscious thought. Stevensons work is taken up by Billett (2001) who combines this approach
with other areas of cognitive psychology to explain vocational performance. For instance, one
area, known as procedural knowledge, comprises techniques, skills, and the know-how to
secure goals (Billett, 2001). As outlined in Table 8, procedural knowledge is proposed to
consist of three levels (Stevenson, 1994).

Table 8 Categories of higher order procedures

Modified from Stevenson (1994, pp. 13-14)

1st-order procedures

Procedures for the achievement of specific goals

2nd-order procedures

Procedures which operate on specific procedures to achieve more

general goalschecking and evaluating one's own cognitive
actions and solving problems by dis-aggregating them into subproblems; finding applicable specific procedures; sequencing them
and evaluating how well they will solve the new problem.

3rd-order procedures

The cognitive switching between first and second order

procedures.

142

Chapter Six
Like walking, 1st-order procedures require little if any conscious thought. Billett (2001, p. 23)
explains that as procedures and concepts are practised, they are transformed into smooth
singular operations: a process called compilation. Billetts work reflects that of Glaser (1991),
who describes how skills are chunked into ever increasingly larger forms. One of the unique
features of 1st-order procedures is that they can be performed with little, if any, conscious
effort, leaving more cognitive capacity for monitoring other areas. The process of compilation
can explain how individuals can develop and carry out such complex tasks as flying. Though
no apparent conscious effort is required, monitoring of 1st-order procedures is continually
strengthening and modifying these tasks. This enables individuals to perform more tasks at
once. If tasks are novel or new, 2nd-order procedures are required to complete tasks, and this
process is far more effortful and cognitively taxing (Billett, 2001).

As the current study has determined, there is a necessity for pilots to maintain essential skills
during flight, such as SA. It could be argued that those pilots who have many skills at a 1storder level will have greater cognitive capacity to deal with difficult scenarios. For this reason,
it is suggested that training programs that focus on developing as many skills at a 1st-order
level as possible, may improve pilot performance. Such programs would involve the modelling
of complex skills (Billett, 2001), rehearsal, and feedback (Henley, Wiggins, Bye, & Turney,
2003), such that pilots are able to develop skills that they can implement automatically. It is
argued that pilots whose skills are more developed, and operating at a 1st-order level, will have
greater capacity to maintain SA during complex and demanding scenarios.

It comes as no surprise that training which improves SA is not implemented universally in

airlines (Endsley & Robertson, 2000), when reflecting on an industry that continues to grapple
with the vast amounts of research into NTS, which leaves individual airlines to identify their
own focus areas for training. The current study may assist airlines with honing this focus.
143

Chapter Six
Training programs that direct pilots to improve perception, combined with methods used to
reduce level 1 errors, and improve the development of 1st-order procedures, could assist in
improving pilots SA overall.

6.4.5.3 Increased emphasis on management principles during training of airline captains

The final implication for this study is in relation to management training. Specifically, it was
identified that management of crew activities is seen as an enabling skill. In particular, the
findings identified that utilising effective management strategies is one way of maintaining and
improving SA (Section 5.3.5.1). These findings are supported by research conducted by
Schutte & Trujillo (1996), who found that pilots who have effective task management
strategies are more likely to maintain SA. As identified in Section 2.4.4, there is limited
research that identifies how pilots, or captains, are to manage effectively. Nor are training
programs commonly provided to enable captains to achieve best practice as a manager. Rather,
management practices appear to be learnt largely through the process of trial and error. This
approach to learning, may create an unnecessary environment where higher levels of
inconsistent approaches to management occur on the flight deck of aircraft.

Recommendations
Senior managers within airline training departments should conduct a review of their
training practices to identify if they align with the essential skills identified in the
current study. This process may highlight whether particular skills are over or under
emphasised;
Senior managers within airline training departments should investigate whether their
current programs emphasise to pilots the importance of SA, how to identify when SA is
reduced, and methods of regaining SA; and

144

Chapter Six
Further research should be undertaken to identify effective methods to manage airline
operations in order to improve the way that management skills are taught to pilots and,
in particular, captains within airlines.

6.4.6 Current policy in aviation

Many of the implications that have been outlined above point towards improvements in pilots
assessment, training for check captains, and improved training methods for pilots. Although
these implications may well assist individual airlines, the regulatory authorities, tasked with the
safety oversight, must be well informed about those practices occurring within the airlines.
Recently, the UK Civil Aviation Authority (CAA) identified that the main causal factors in
aviation accidents between 1997 and 2006 were the omission of action/inappropriate action,
flight handling and, finally, lack of positional awareness - in air, with each contributing to
accidents 39%, 29% and 27% respectively, or a significant 95% of total accidents (CAA, 2008,
p. 2). These causal factors can be categorised in to decision making, flying, and SA skills, all
of which have been identified as essential skills within the current study.

The current study identified that check captains tasked to assess pilots performance are aware
of the importance of both technical skills and NTS. These check captains assess both technical
skills and NTS during performance assessment. CASA continues to enforce the assessment of
technical skills, and supports the assessment of NTS, with the introduction of recent policy
(CASA, 2009d). However, the policy does not clearly state whether technical skills should
continue to be the focus of assessment. This research demonstrates that technical skills and
NTS must be integrated into a more holistic approach to assessment, where all essential criteria
are given emphasis, rather then technical skills alone.

145

Chapter Six
Recommendations
It is recommended that regulatory authorities continue to closely follow research into
assessment of technical skills and NTS. Additionally, it is suggested that they may
begin investigating ways to integrate assessment criteria, as outlined in the
recommendations of Section 6.2.2. Furthermore, during the merging of technical skills
and NTS, effective verbal descriptors should be evaluated and, where necessary,
redesigned to assist check captains in assessing NTS.

6.5

Conclusion

This thesis began by discussing the assumptions made by the travelling public about the skills
that pilots have to fly modern jet aircraft. Importantly, the public assume that airline pilots are
capable of performing the tasks associated with flying, in normal and difficult circumstances.
The current study has shown that, although these assumptions may be well founded, the
assessment of pilots performance is somewhat less transparent than would be assumed. To
improve the existing assessment process, this study identifies the criteria that are currently used
to assess pilots performance, and the relationships between these criteria. As a result of the
findings of this study, a model was developed that contained essential and enabling skills,
which allows the process for assessing airline pilots performance to be refined. The model
developed is now being trialled in practice to assist check captains to assess pilots
performance. Given the current regulatory changes, and difficultly with NTS assessment, the
findings of this study have made a timely contribution to pilot assessment. Finally, the findings
have important implications for the profession of aviation and the wider professional
community.

146

References

References
Abbott, T. S. (1993). Functional categories for future flight deck designs. Unpublished
manuscript. Hampton, Virginia.
Air Accident Investigation Branch (1990). Report on the accident to Boeing 737-400, GOBME, near Kegsworth, Leicestershire on 8 January 1989 (No. 4/1990). London.
Amalberti, R. (2001). The paradoxes of almost totally safe transportation systems. Safety
Science, 37, 109-126.
Amalberti, R., & Deblon, F. (1992). Cognitive modelling of fighter aircraft process control: A
step towards an intelligent on-board assistance system. International Journal of ManMachine Systems, 36(5), 639-671.
Australian Government (1988). Civil Aviation Act 1988. Canberra.
Australian Transport Safety Bureau (1995). Fairchild Industries Inc SA227-AC, VH-NEJ,
Tamworth, NSW 16 September 1995. Canberra.
Australian Transport Safety Bureau (2000). Beech Aircraft Corp 1900D, VH-NTL. Canberra.
Australian Transport Safety Bureau (2006). Aircraft Loss of Control, Lake George, NSW; VHTAG, SA227-AC Metro III. Canberra.
Australian Transport Safety Bureau (2009). Fuel starvation Jundee Airstrip, WA - 26 June
2007 VH-XUE Empresa Brasileira de Aeronutica S.A., EMB-120ER. Canberra.
Babbie, E. (1994). The practice of social research. Melbourne: Wadsworth Publishing
Company.
Baker, D. P., & Dismukes, R. K. (1999). Training pilot instructors to assess CRM: The utility
of frame-of-reference (FOR) training. In R. Jensen (Ed.), Proceedings of the 10th
International Symposium on Aviation Psychology (pp. 291-300). Columbus, OH: Ohio
State University.
Baker, D. P., & Dismukes, R. K. (2002). A framework for understanding crew performance
assessment issues. The International Journal of Aviation Psychology, 12(3), 205-222.
Banbury, S., Dudfield, H., Hrmann, H., & Henning, S. (2002). FASA: Development and
validation of a novel measure to assess the effectiveness of commercial airline pilot
situational awareness training. The International Journal of Aviation Psychology, 17(2),
131-152.
Barnacle, R. (2001). Introduction. In R. Barnacle (Ed.), Phenomenology (pp. v-x). Melbourne:
RMIT University Press.
Beach, L. R., & Lipshitz, R. (1993). Why classical decision theory is an inappropriate standard
for evaluating and aiding most human decision making. In G. A. Klein, J. Orasanu, R.
147

References
Calderwood & C. Zsambok (Eds.), Decision making in action: Models and methods
(pp. 21-31). Norwood, New Jersey: Ablex Publishing Corporation.
Beaubien, J. M., Baker, D. P., & Salvaggio, A. N. (2004). Improving the construct validity of
line operational simulation ratings: Lessons learned from the assessment center. The
International Journal of Aviation Psychology, 14(1), 1-17.
Billett, S. (2001). Learning in the workplace: Strategies for effective practice. Sydney: Allen &
Unwin.
Billett, S., & Stevenson, J. (2000). Curriculum theory & design. Unpublished manuscript,
Brisbane.
Black, L.L. (2009). Clinical supervision for international counsellors-in-training: Implications
for supervisors. Journal of professional counselling: Theory, and research, 37(2), 5265.
Boeing

(2002).
Accident
rate
history.
Retrieved
24/11/09,
2009,
http://www.boeing.com/commercial/safety/pf/pf_accident_rate_history_cht.html

from

Boeing

(2004). Current market outlook. Retrieved

http://www.boeing.com/commercial/cmo/1-2.html

from

23/06/07,

2007,

Boeing (2009). Next-generation 737 flight deck. Retrieved 01/09/09, 2009, from
http://www.boeing.com/commercial/gallery/737900-11.html
Brannick, M. T., Prince, C., & Salas, E. (2002). The reliability of instructor evaluations of crew
performance: Good news and not so good news. The International Journal of Aviation
Psychology, 12(3), 241-261.
Brearley, L. (2001). Exploring creative forms within phenomenological research. In R.
Barnacle (Ed.), Phenomenology (pp. 74-86). Melbourne: RMIT University Press.
Burns, D. & Walker, M. (2005). Feminist methodologies. In B. Somekh & C. Lewin (Eds.),
Research methodologies in the social sciences (pp. 6673). London: Academic Press
Ltd.
Calkins, S. and Light, G. (2008). University Teaching. In T. L. Good Vol. 2. (Ed), 21st Century
Education: A Reference Handbook (pp445-454). Thousand Oaks, CA: Sage
Publications Inc.
Camic, P. M., Rhodes, J. E., & Yardley, L. (2003). Naming the stars: Integrating qualitative
methods into psychological research. In P. M. Camic, J. E. Rhodes & L. Yardley (Eds.),
Qualitative Research in Psychology (pp. 3-15). Washington, DC: American
Psychological Association. Chute, R. D., & Weiner, E. L. (1995). Cockpit-Cabin
communication: A Tale of Two Cultures. The International Journal of Aviation
Psychology, 5(3), 257-276.
Camp. D.G. (2007). Where do standards come from? A phenomenological study of the
development of national board early childhood/generalist standards. Journal of
research in childhood education, 21(4), 420-437.

148

References
Civil Aviation Authority (2008). Global Fatal Accident Review 19972006 CAP 776. West
Sussex.
Civil Aviation Safety Authority (1988). Civil Aviation Regulation (5.105-5.106). Canberra:
Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2000). Air transport pilot (aeroplane) licence aeronautical
knowledge syllabus. Canberra: Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2002). Advisory Circular AC 121A-09(0) Human Factors and
Crew Resource Management (CRM) Training: [DRAFT]. Canberra.
Civil Aviation Safety Authority (2004a). CAO 45.0 flight crew standards - synthetic trainers.
Canberra: Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2004b). Conditions on air transport pilot (aeroplane)
licences. Canberra: Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2008a). 7.12.2.1.3 Training the Training and Checking Pilots.
In CASA (Ed.), Air operator certification manual. Canberra: Legislative Drafting
Branch, CASA.
Civil Aviation Safety Authority (2008b). CAO 40.2.1 Instrument ratings. Canberra: Legislative
Drafting Branch, CASA.
Civil Aviation Safety Authority (2008c). Day VFR syllabus. Canberra: Legislative Drafting
Branch, CASA.
Civil Aviation Safety Authority (2008d). Implementation of safety management systems (SMS)
and introduction of human factors (HF) training and non-technical skills (NTS)
assessment. Canberra: Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2009a). CAO 40.1.0 Aircraft endorsements-aeroplanes.
Canberra: Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2009b). CAO 82.5 Conditions on air operators certificates
authorising regular public transport operations in high capacity aircraft. Canberra:
Legislative Drafting Branch, CASA.
Civil Aviation Safety Authority (2009c). CASA style of exam questions. Retrieved 14/11/09,
from
CASA:
http://www.casa.gov.au/scripts/nc.dll?WCMS:STANDARD::pc=PC_90082
Civil Aviation Safety Authority (2009d). Implementation of safety management systems (SMS)
and introduction of human factors (HF) training and non-technical skills (NTS)
assessment. Canberra.
Civil Aviation Safety Authority (2009e). More time for human factors training, The CASA
briefing.
Retrieved
01/02/10,
2010,
from
http://www.casa.gov.au/scripts/nc.dll?WCMS:STANDARD::pc=PC_93491.

149

References
Civil Aviation Safety Authority (2010). Non-technical skills training and assessment for
regular public transport operation - DRAFT. Canberra. Legislative Drafting Branch,
CASA.Cohen, M. S. (1993). The naturalistic basis of decision bias. In G. A. Klein, J.
Orasanu, R. Calderwood & C. Zsambok (Eds.), Decision making in action: models and
methods (pp. 51-91). Norwood, NJ: Ablex Publishing Corporation.
Cohen, L., & Manion, L. (1994). Research methods in education. London: Routledge.
Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education. New York:
Routledge Falmer.
Cooksey, R. W. (1996). Judgement analysis: Theory, methods, and applications. Armidale:
Academic Press.
Crotty, M. (1998). The foundations of social research: Meaning and perspective in the
research process. St Leonards: Allen & Unwin Pty Ltd.
DallAlba, G. (2002). Understanding medical practice: Different outcomes of a pre-medical
program. Advances in Health Sciences Education, 7, 163-177.
Dall'Alba, G. (2009). Learning to Be Professionals. Dordrecht, NL: Springer.
DallAlba, G., & Sandberg, J. (2006). Unveiling professional development: A critical review of
stage models. Review of Educational Research, 76(3), 383-412.
de Montalk, R. (2008). Developing proficiency in air transport pilots: The case for the
introduction of non-technical skills in basic flight training programmes. Massey
University, Palmerston North.
Dewey, J. (1938). Experience and education. New York: Macmillan.
Dreyfus, H. L., Dreyfus, S. E., & Athanasiou, T. (1986). Mind over machine: The power of
human intuition and expertise in the era of the computer. New York: Free Press.
Durso, F. T., & Gronlund, S. D. (1999). Situational awareness. In F. T. Durso (Ed.), Handbook
of applied cognition (pp. 283-314). West Sussex: John Wiley & Sons Ltd.
Edkins, G. D. (2005). A review of the benefits of aviation human factors training. In D. Harris
& H. C. Muir (Eds.), Contemporary issues in human factors and aviation safety (pp.
117-131). Aldershot: Ashgate.
Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement.
Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 97-101). Santa
Monica, CA: Human Factors Society.
Endsley, M. R. (1994). Situational awareness in dynamic human decision making:
Measurement. In R. D. Gilson, D. J. Garland & J. M. Koone (Eds.), Situational
awareness in complex systems (pp. 79-97). Daytona Beach, Florida: Embry-Riddle
Aeronautical University Press.
Endsley, M. R. (1998). Design and evaluation for situation awareness enhancements. Paper
presented at the Human Factors Society 32nd annual meeting, Santa Monica, CA.
150

References
Endsley, M. R. (1999). Situational awareness in aviation systems. In D. J. Garland, J. A. Wise
& V. D. Hopkin (Eds.), Handbook of aviation human factors (pp. 257-276). Mahwah,
NJ: Lawrence Erlbaum Associates.
Endsley, M. R. (2010). Situational awareness in aviation systems. In J. A. Wise, V. D. Hopkin
& D. J. Garland (Eds.), Handbook of aviation human factors (Vol. 2, pp. 12/11-22).
London: CRC Press.
Endsley, M. R., & Robertson, M. M. (2000). Training for situational awareness in individuals
and teams. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and
measurement (pp. 349-366). Mahwah, NJ: Lawrence Erlbaum Associates.
Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of
maximal adaptation to task constraints. Annual Review of Psychology, 47(February),
273-305.
Evans, G. (1994). Learning in apprenticeship courses. In J. Stevenson (Ed.), Cognition at
work: The development of vocational expertise (pp. 76-102). Leabrook SA: National
Centre for Vocational Education Research Ltd (NCVER).
Flick, U. (2006). An introduction to qualitative research: SAGE, London.
Flin, R., Fletcher, G., McGeorge, P., Glavin, R., Maran, N., & Patey, R. (2003a, October).
Rating anaesthetists' non-technical skills - The ANTA system. Paper presented at the
47th Human Factors and Ergonomics Society conference, Denver.
Flin, R., Martin, L., Goeters, K., Hrmann, H., Amalberti, R., Valot, C., et al. (2003b).
Development of the NOTECHS (non-technical skills) system for assessing pilots' skills.
Human Factors and Aerospace Safety, 3(2), 97-119.
Flin, R., Goeters, K., Hrmann, H., & Martin, L. (1998, September). A generic structure of
non-technical skills for training and assessment. Paper presented at the 23rd conference
of the European Association for Aviation Psychology, Vienna.
Flin, R., & Maran, N. (2004). Identifying and training non-technical skills for teams in acute
medicine. Quality and Safety in Health Care, 13(1), 80-84.
Flin, R., & Martin, L. (2001). Behavioural markers for crew resource management: A review
of current practice. The International Journal of Aviation Psychology, 11(1), 95-118.
Flin, R., Martin, L., Goeters, K., Hrmann, H., Amalberti, R., Valot, C., et al. (2005).
Development of the NOTECHS (non-technical skills) system for assessing pilots' CRM
skills. In D. Harris & H. C. Muir (Eds.), Contemporary issues in human factors and
aviation safety (pp. 133-154). Aldershot: Ashgate.
Flin, R., O'Connor, P., & Crichton, M. (2009). Safety at the sharp end. Aldershot: Ashgate.
Foddy, W. (1995). Constructing questions for interviews and questionnaires. Melbourne:
Cambridge University Press.

151

References
Foushee, H. C., & Helmreich, R. L. (1988). Group interaction and flight crew performance. In
E. L. Weiner & D. C. Nagel (Eds.), Human factors in aviation (pp. 189-227). Los
Angeles: Academic Press.
Freebody, P. (2003). Qualitative research in education: Interaction and practice. London:
Sage.
Gay, L. R., & Airasian, P. (2000). Educational research: Competencies for analysis and
application. Sydney: Prentice-Hall.
Giorgi, A. (1994). A phenomenological perspective on certain qualitative research methods.
Journal of Phenomenological Psychology, 25(2), 190-220.
Giorgi, A. (1997). The theory, practice, and evaluation of the phenomenological method as a
qualitative research procedure. Journal of Phenomenological Psychology, 28(2), 235260.
Giorgi, A., & Giorgi, B. (2003). The descriptive phenomenological psychological method. In
P. M. Camic, J. E. Rhodes & L. Yardley (Eds.), Qualitative research in psychology (pp.
243- 273). Washington, DC: American Psychological Association.
Giorgi, A., & Giorgi, B. (2008). Phenomenology. In J. A. Smith (Ed.), Qualitative psychology
(pp. 26-52). London: Sage Publications Ltd.
Glaser, R. (1991). Expertise and Assessment. In M. C. Wittrock & E. L. Baker (Eds.), Testing
& Cognition (pp. 17-30). Englewoods Cliffs, NJ: Prentice Hill.
Goldsmith, T. E., & Johnson, P. J. (2002). Assessing and improving evaluation of aircrew
performance. The International Journal of Aviation Psychology, 12(3), 223-240.
Hartel, C., Smith, K., & Prince, C. (1991, April). Defining aircrew situational awareness:
Searching for mishaps and meanings. Paper presented at the 6th International
Symposium on Aviation Psychology, Columbus, OH.
Hawkins, F. H. (1987). Human factors in flight. Vermont: Ashgate Publishing Limited.
Hayes, J. R. (1989). Complete problem solver. Philadelphia: Franklin Institute Press.
Hein, S. F., & Austin, W. J. (2001). Empirical and hermeneutic approaches to
phenomenological research in psychology: A comparison. Psychological Methods,
6(1), 3-17.
Helmreich, R. L. (1995). Commission of inquiry into the Air Ontario crash at Dryden, Ontario.
The CRM Advocate.
Helmreich, R. L. (2000). On error management: Lessons from aviation. British Medical
Journal, 320(7237), 781-786.
Helmreich, R. L., & Foushee, H. C. (1993). Why crew resource management? Empirical and
theoretical bases of human factors in aviation. In E. L. Weiner, B. G. Kanki & R. L.
Helmreich (Eds.), Cockpit resource management (pp. 3-46). San Diego: Academic
Press.
152

References
Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The evolution of crew resource
management training in commercial aviation. The International Journal of Aviation
Psychology, 9(1), 19-32.
Henderson, S. (2009). Frames in the flight deck: A sociological approach to situational
awareness. University of New South Wales, Sydney.
Holt, R. W., Hansberger, J. T., & Boehm-Davis, D. A. (2002). Improving rater calibration in
aviation: A case study. The International Journal of Aviation Psychology, 12(3), 305330.
Husserl, E. (1970). Logical investigation / Edmund Husserl (2nd ed. J. N. Findlay, Translated).
London: Humanities Press.
International Airline Transport Association (2008). 2008 aviation safety performance fact
sheet. Retrieved from http://www.iata.org/NR/rdonlyres/789D1FAC-895E-49CE-93204ACDFAA29672/0/2008AviationSafetyperformance.pdf
International Civil Aviation Organisation. (1998). Human factors training manual: Doc 9683AN/950. Montreal: ICAO.
International Civil Aviation Organisation (1999). Plan of action for the flight safety and human
factors programme 2000-2004 (AN-WP/7462). ICAO, Montreal.
International Civil Aviation Organisation (2004). Plan of action for the flight safety and human
factors programme 2005-2009 (AN-WP/7956). ICAO, Montreal.
International Civil Aviation Organisation (1990). Humans factors the focus of milestone
Leningrad seminar. ICAO Journal, 45(5), 13-16.
International Airline Transport Association (2009). IATA training and qualification initiative
(ITQI). Montreal.
International Civil Aviation Organisation (2009). The convention on international civil aviation
annexes
1
to
18.
Retrieved
03/09/09,
2009,
from
http://www.icao.int/icaonet/anx/info/annexes_booklet_en.pdf
International Civil Aviation Organisation (2010). Catalogue of ICAO publications.
Retrieved
18/09/10,
2010,
http://www.icao.int/icao/en/sales/cat_2009_2010_revised_edition_en.pdf

from

Janis, A., Cannon-Bowers, & Salas, E. (1998). Individual and team making under stress:
Theoretical underpinnings. In A. Janis & E. Salas (Eds.), Making decisions under
stress: Implications for individual and team training (pp. 17-38).Washington:
American Psychology Association.
Johnston, N. (1993). CRM: Cross-cultural perspectives. In E. Wiener, B. Kanki, & R.
Helmreich (Eds.), Cockpit Resource Management (pp. 367-398). San Diego, CA:
Academic Press.
Johnston, A. N. & Maurino, D.E. (1990). Human factors training for aviation personal. ICAO
Journal, 45(5), 16-19.Johnston, A. N., Rushby, N., & Maclean, I. (2000). An assistant
153

References
for crew performance assessment. The International Journal of Aviation Psychology,
10(1), 99-108.
Joint Aviation Authorities (2002). Final report JAR TEL consolidation of results.
Kanki, B. G., & Palmer, M. T. (1993). Communication and crew resource management. In E.
L. Weiner, B. G. Kanki & R. L. Helmreich (Eds.), Cockpit resource management (pp.
99-135). San Diego: Academic Press.
Kenny, A. (2007). Philosophy in the modern world. Oxford: Oxford University Press.
Kern, T. (1997). Redefining airmanship. New York: McGraw Hill.
Klapproth, F. (2008). Time and decision making. Cognitive, Affective & Behavioural
Neuroscience, 8(4), 509-524.
Klein, G. A. (1993). A recognition-primed decision (RPD) model of rapid decision making. In
G. A. Klein, J. Orasanu, R. Calderwood & C. Zsambok (Eds.), Decision making in
action: models and methods (pp. 138-147). Norwood, New Jersey: Ablex Publishing
Corporation.
Klein, G. A. (2008). Naturalistic decision making. Human Factors, 50(3), 456-460.
Kvale, S. (1995). The social construction of validity. Qualitative Inquiry, 1(1), 19-40.
Kvale, S. (1999). The psychoanalytical interview as qualitative research. Qualitative Inquiry,
5(1), 87-113.
Kvale, S. (2001). The psychoanalysis interview as qualitative research. Psychologische
Beitrge, 43.
Kvale, S. (2006). Dominance through interviews and dialogues. Qualitative Inquiry, 12(3),
480-500.
Lane, S. & Tierney, S.T. (2008). Performance Assessment. In T. L. Good Vol. 1 (Ed), 21st
Century Education: A Reference Handbook (pp461-470). Thousand Oaks, CA: Sage
Publications Inc.
Latham, G. (2001). A journey towards catching phenomenology. In R. Barnacle (Ed.),
Phenomenology (pp. 41-57). Melbourne: RMIT University Press.
Lave, J. (1988). Cognition in practice: Mind, mathematics and culture in everyday life.
Cambridge: Cambridge University Press.
Lee, A. T. (2005). Flight Simulation: Virtual environments in aviation. Hampshire: Ashgate.
Lehman, C. (2009). An abundance of intellect. The Journal for Civil Aviation Training (3), 5.
Lempereur, I., & Lauri, M. A. (2006). The psychological effects of constant evaluation on
airline pilots: An exploratory study. The International Journal of Aviation Psychology,
16(1), 113-133.

154

References
LePlat, J. (1991). Organisation of activity in collective tasks. In B. Rasmussen, B. Brehmer &
J. LePlat (Eds.), Distributed decision making: Cognitive models for cooperative work
(pp. 51-73). Chichester: John Wiley.
Linn, R. L., & Gronlund, N. E. (1995). Measurement and assessment in teaching. Englewood
Cliffs: N.J. Merrill.
Maurino, D.E. (1994). ICAO workshops gather important knowledge on perception of human
factors training. ICAO Journal, 53(5), 17-29.
Maurino, D.E. (1995). ICAO annex amendment introduces mandatory human factors training
for airline flight crews. ICAO Journal, 50(7), 13, 24-25.
Maurino, D.E. (1998). ICAO supports proactive approach to managing human factors issues
related to advanced technology. ICAO Journal, 42(5), 13-16.
Maurino, D. E., & Murray, P. (2010). Crew resource management - A time for reflection. In J.
A. Wise, V. D. Hopkin & D. J. Garland (Eds.), Handbook of aviation human factors
2nd Edition (pp. 10.1-10.20). Boca Raton: CRC Press.
Mavin, T. J., & Dall'Alba, G. (2009, July). Identifying minimum standards of non-technical
skills in airline pilots undergoing command promotion. Paper presented at the Aviation
Education and Research conference, Blenheim, NZ.
Mavin, T. J., & Dall'Alba, G. (2010, April). A model for integrating technical skills and NTS in
assessing pilots performance. Paper presented at the 9th International Symposium of
the Australian Aviation Psychology Association, Sydney.
Mavin, T. J., & Murray, P. (2010). The development of airline pilot skills through practice. In
S. Billett (Ed.), Learning through practice: Models, traditions, orientations and
approaches (pp. 268-286). Dordrecht, NL: Springer.
McKenna, W. (1982). Husserl's "Introductions to Phenomenology" Interpretations and
Critique. The Hague: Martinus Nijhoff Publisher.
Mercer, J. (2007). The challenge of insider research in educational institutions: wielding a
double-edged sword and resolving delicate dilemmas. Oxford Review of Education,
33(1), 1-17.
Miles, M. B. (1979). Qualitative data as an attractive nuisance: The problem of analysis.
Administrative Science Quarterly, 24(4), 590-601.
Mondey, D. (Ed.). (1977). The international encyclopaedia of aviation. London: Octopus
Books Ltd.
Moore, B. (2004). The Australian Oxford dictionary. Melbourne: Oxford University Press.
Moran, D. (2000). Introduction to phenomenology: Routledge Abingdon, Oxon.
Moroney, W. F., & Moroney, B. W. (1999). Flight simulation. In D. J. Garland, J. A. Wise &
V. D. Hopkin (Eds.), Handbook of aviation human factors (pp. 355-388). London:
Lawrence Erlbaum Associates.
155

References
Mulqueen, C., Baker, D. P., & Dismukes, R. K. (2002). Pilot instructor rater training: The
utility of the multifacet item response theory model. The International Journal of
Aviation Psychology, 12(3), 287-303.
Munro, P. A., Kanki, B. G., & Jordan, K. (2008). Beyond "Inop": Logbook Communication
Between Airline Mechanics and Pilots. The International Journal of Aviation
Psychology, 18(1), 86-103.
Nagel, D. C. (1988). Human error in aviation operations. In E. L. Wiener & D. C. Nagel (Eds.),
Human factors in aviation (pp. 263303). London: Academic Press Ltd.
O'Connor, P., Hrmann, H. J., Flin, R., Lodge, M., & Goeters, K.-M. (2002). Developing a
method for evaluating crew resource management skills: A European perspective. The
International Journal of Aviation Psychology, 12, 263-285.
O'Connor, P., Campbell, J., Newmon, J., Melton, J., Salas, E., & Wilson, K. A. (2008). Crew
resource management training effectiveness: A meta-analysis and some critical needs.
The International Journal of Aviation Psychology, 18(4), 353-368.
Olson, W. A., & Sarter, N. B. (2000). Automation management strategies: Pilot preferences
and operational experiences. The International Journal of Aviation Psychology, 10,
327-341.
Orasanu, J. (1993). Decision making in the cockpit. In E. L. Weiner, B. G. Kanki & R. L.
Helmreich (Eds.), Cockpit resource management (pp. 137-172). San Diego: Academic
Press.
Orasanu, J., & Connolly, C. (1993). The reinvention of decision making. Norwood, New
Jersey: Ablex Publishing Corporation.
Robbins, S.P., Bergman, R., Stagg, I., & Coulter, M. (2009). Foundations of management (5th
ed.). Sydney: Pearson Education Australia.
Rolfe, G. (2006). Validity, trustworthiness and rigour: Quality and the idea of qualitative
research. Journal of Advanced Nursing, 53(3), 304-310.
Roscoe, S. N. (1980). Concepts and definitions. In S. N. Roscoe (Ed.), Aviation psychology
(pp. 1-10). Ames, IOWA: IOWA State University Press.
Roscoe, S. N. (1984). Transfer and cost effectiveness of ground-based flight trainers. Aviation
psychology (pp. 197-203). Iowa: State university press.
Sadler, R. D. (1987). Specifying and promulgating achievement standards. Oxford Review of
Education, 13(2), 191-209.
Sadler, R. D. (1989). Formative assessment and the design of instructional systems.
Instructional Science, 18, 119-144.
Sadler, R. D. (1998). Formative assessment: Revisiting the territory. Assessment in Education,
5(1), 77-84.
Sadler, R. D. (2005). Interpretations of criterion-based assessment and grading in higher
education. Assessment & Evaluation in Higher Education, 30(2), 175-194.
156

References
Sadler, R. D. (2009). Grade integrity and the representation of academic achievement. Studies
in Higher Education, 34(7), 807-826.
Salas, E., Bowers, C. A., & Prince, C. (1998). Special issues on simulation and training in
aviation. The International Journal of Aviation Psychology, 8(3), 195-196.
Salas, E., Burke, C. S., & Stagl, K. C. (2004). Developing teams and team leaders: Strategies
and principles. In D. V. Day, S. J. Zaccaro & S. M. Halpin (Eds.), Leader development
for transforming organizations (pp. 325-355). New Jersey: Lawrence Erlbaum
Associates.
Salas, E., Rhodenizer, L., & Bowers, C. A. (2000). The design and Delivery of Crew Resource
Management Training: Exploiting Available Resources. Human Factors, 42(3), 490511.
Salas, E., Wilson, K. A., Burke, C. S., Wightman, D. C., & Howse, W. R. (2006). A checklist
for crew resource management training. Ergonomics and Design, 14(2), 6-15.
Sandberg, J. (1994). Human competence at work: An interpretative approach. Goteborg,
Sweden: BAS.
Sandberg, J. (1997). Are phenomenographic results reliable? Higher Education Research &
Development, 16(2), 203-212.
Sandberg, J. (2000). Understanding human competence at work: An interpretive approach.
Academy of Management Journal, 43, 9-25.
Sandberg, J. (2001). The constructions of social constructionism. In S. E. Sjstrand & J.
Sandberg (Eds.), Invisible management: The social construction of leader-ship (pp. 2848). London: Thomson.
Sandberg, J. (2005). How do we justify knowledge produced within interpretive approaches?
Organisational Research Methods, 8(1), 41-68.
Schn, D. A. (1983). The reflective practitioner. London: Maurice Temple Smith Ltd.
Schutte, P. C., & Trujillo, A. C. (1996). Flight crew task management in non-normal situations.
Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting.
Santa Monica, CA.
Seamster, T. L., Edens, E. S., & Holt, R. W. (1995). Scenario event sets and the reliability of
CRM assessment. Paper presented at the 8th International Symposium on Aviation
Psychology, Columbus, OH.
Silverman, D. (2001). Interpreting qualitative data: Methods for analysing talk, text and
interaction (2nd ed.). London: Sage Publications.
Smith, J. A., & Osborn, M. (2008). Interpretive phenomenological analysis. In J. A. Smith
(Ed.), Qualitative psychology: A practical guide to research methods (pp. 53-80).
London: SAGE.

157

References
Stevenson, J. (1994). Vocational expertise. In J. Stevenson (Ed.), The development of
vocational expertise (pp. 7-35). Leabrook SA: National Centre for Vocational
Education Research (NCVER).
Stevenson, J. (2003). Expertise for the workplace. In J. Stevenson (Ed.), Developing vocational
expertise (pp. 3-25). Crows Nest: Allen & Unwin.
Svatek. (1990). Techniques in CRM training. ICAO Journal, 45(10), 12-13.
Taggart, W.R. (1987). CRM - a different approach to human factors training. ICAO Journal,
42(5), 13-16.
Thomas, M. J. W. (2003). Improving organisational safety through the integrated evaluation of
operational and training performance: an adaptation of the line operational safety audit
(LOSA) methodology. Human Factors and Aerospace Safety, 3(1), 25-45.
Thomas, M. J. W. (2004). Predictors of threat and error management: Identification of core
nontechnical skills and implications for training systems design. The International
Journal of Aviation Psychology, 14(2), 207-231.
van Avermaete, J. A. G. (1998). NOTECHS; Non-technical skills evaluation in JAR-FCL.
Amsterdam: national Aerospace Laboratory.
Webster-Wright, A. (2006). Reframing professional development through understanding
authentic professional learning. University of Queensland, Brisbane.
Welch, A. J. (2001). Finding a way through the maze. In R. Barnacle (Ed.), Phenomenology
(pp. 58-73). Melbourne: RMIT University Press.
Weisman, G. (1991). A new partnership in CRM training. ICAO Journal, 46(10), 10-12.
West, M. A. (2004). Effective teamwork: Practical lessons from organizational research (2nd
ed.). Leicester: BPS Blackwell.
Wickens, C. D. (1999). Cognitive factors in aviation. In F. T. Durso (Ed.), Handbook of
applied cognition (pp. 247-282). West Sussex: Wiley.
Wiggins, M., Connan, N., & Morris, C. (1996). Weather related decision making and selfperception amongst pilots. In B. J. Hayward & A. R. Low (Eds.), Applied aviation
psychology (pp. 193-200). Aldershot, England: Ashgate Publishing Ltd.
Wilson, K. A., Guthrie, J. W., Salas, E., & Howse, W. R. (2010). Team process. In J. A. Wise,
V. D. Hopkins & D. J. Garland (Eds.), Handbook of aviation human factors (Vol. 2, pp.
9/1-22). London: CRC Press.
Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative
review. Journal of Occupational and Organizational Psychology, 67(3), 189-205.
Yardley, L. (2008). Demonstrating validity in qualitative psychology. In J. A. Smith (Ed.),
Qualitative psychology: A practical guide to research methods (pp. 235-251). London:
Sage.

158

Appendix 1

Appendix 1 Technical skills assessment forms

159

Appendix 1

160

Appendix 1

161

Appendix 1

162

Appendix 2

Appendix 2 Non-technical skills assessment

Non-Technical Skills - Generic structure and word pictures of elements for good and poor performance (Flin et al., 2003b)
Category

Element
Team building
Considering others

Co-operation
Supporting other
Conflict solving

Use of authority

Maintaining standards

Leadership &
Management

Planning and coordination

Workload management

Poor

Good

Blocks open communication

Keeps barriers between CM
Competes with other CM
Ignores suggestions of other CM
Does not take account of the condition of other CM
Shows no reaction to other CM
Hesitates to help other CM in demanding situations
Does not offer assistance

Establishes atmosphere for open communication

Encourages inputs and feedback from others
Does not compete with others
Takes notice of suggestion of other CM even if they do not agree
Takes account of other CM condition
Gives personal feedback
Helps other CM in demanding situation
Offers assistance

Overreacts in interpersonal conflicts

Sticks to own position without considering a compromise
Accuses other CM of making errors
Hinders or withholds crew involvement
Passive, does not show initiative for decisions, own position
not recognisable
Ignores suggestions of others
Does not comply with SOPs, does not monitor crew for SOP
compliance
Does not intervene in case of deviations
Deviation from SOPs are neither announced nor consulted
Does not care for performance affectivity
Plans only for him/herself, crew not involved
Intentions not stated or confirmed
Changes plan without informing crew or follows plan blindly
Goals and boundaries remain unclear
Flying solo without other crew members involvement
Secondary operational tasks interfere with the primary flight
duties
Workload is increased through inadequate planning
Ignores signs of stress and fatigue

Keeps calm in interpersonal conflicts

Suggests conflict solution
Concentrates on what is right rather than who is right
Takes initiative to ensure crew involvement and task completion
Takes command if situation requires, advocates own position
Motivates crew by appreciation and coaches when necessary
Subscribes to SOPs, makes sure SOP compliance in crew
Intervenes if task completion deviates from standard if necessary
With crew being consulted, deviates from SOPs if necessary
Demonstrates will to achieve top performance
Encourages crew participation in planning and task completion
Plan is clearly stated and confirmed
With crew being consulted, changes plan if necessary
Clearly states goals and boundaries for task completion
Distributes tasks among the crew, checks and corrects
appropriately
Secondary operational tasks are prioritised to retain sufficient
resources for primary flight duties
Allots adequate time to complete tasks
Notifies signs of stress and fatigue

163

Appendix 2
Unaware of system changes
Does not signal awareness of changing systems

Monitors and reports changes in systems states

Acknowledges entries and changes to systems

Does not enquire about environmental changes

Does not comment on relevant environmental factors, or is
surprised by them
Operates a closed shop
Does not set priorities with time limits
Does not discuss relationship between past events and
present/future
Nature of problem not stated or failure to diagnose
No discussion of probable cause

Collects information about environment (poison, weather and

traffic)
Shares key information about environment with crew
Contacts outside resources when needed (to maintain SA)
Discusses time constraints with crew
Discusses contingency strategies
Identifies possible future problems
Gathers information to identify problem
Reviews causal factors with other crew member

Does not search for information

Does not ask for crew alternatives

States alternative options

Asks crew members for options

Risk assessment

Inadequate discussion of limiting factors with crew

No consideration of limiting factors

Considers and shares estimated risks of alternative options

Talks about possible risks for action in terms of crew limits

Option selection

Does not inform crew of decision path being taken

Confirms and states selected option/agreed action

Outcome review

Fails to check selected outcomes against goal

Checks outcomes against plan

Systems awareness

Situational
Awareness

Environmental awareness

Time awareness and

predicting future events
Problem diagnosis
Option generation

Decision
Making

164

Appendix 3

Appendix 3 Participant information and consent forms

Printed on Official University of Queensland paper

Consent Form
Dear Captain

25th June 2008

My name is Timothy John Mavin and I am completing my Doctor of Education degree at the
University Of Queensland School Of Education. I am currently conducting research into the
identification of minimum standards in Airline Pilot non-technical skills. This research has
been approved through the University ethics committee.
By interviewing a number of experienced Check Captains, I plan to investigate their
experiences in assessment of pilots, especially in regard to their initial check as a Captain. I
believe the findings within this study will assist the airline industry to better understand the
standards that have been and are currently applied to pilots within the industry.
The interview will occur in a place that is suitable to yourself. During the interview, you will
be asked to describe some of your experiences as an experienced airline Check Captain. To
assist me in reviewing the information, I am requesting that I am able to audio record the
interviews. At a later date, I may ask you to clarify some items, so that your statements are not
incorrectly interpreted. This can be done at your convenience.
All information that I gain from this interview will, as per university guidelines, be kept in the
strictest confidence. Furthermore, should you wish to withdraw at any time during the research,
you may do so without prejudice. No names of pilots or airlines will be used within the
reporting of this research.
As mentioned, this research has been approved by the University of Queensland. If you would
prefer to talk to an independent person about your involvement, could I direct you to my
principal advisor Dr Gloria DallAlba on (07) 3365 6658 or the School of Education Ethics
Administrative Officer on (07) 3365 6502.
Thanking you for your co-operation.

Tim Mavin

165

Appendix 3
Check Captain Consent form

I have read the accompanying letter explaining the project called identification of minimum
standards in Airline Pilot non-technical skills, which is being conducted by Timothy Mavin
and supervised by Doctor Gloria DallAlba from the University of Queensland.
I understand that:
I will be interviewed by Tim Mavin who may ask me to clarify details at a later date,
Participation is voluntary and that I can withdraw from the study at any time without
prejudice,
No names will be used in any results or publications arising from the study and all
information collected will be securely stored and treated in strict confidence,
I can have access to the information gathered on myself in this project,
I can call Tim Mavin on (07) XXXX XXXX or (0412) XXX XXX and request
information about the project,
I can speak to an officer of the University not involved in the study, by contacting the
School of Education Ethics Administrative Officer on 3365 6500.

I Captain ________________________________________agree to participate in the research

project outlined above and being conducted by Tim Mavin under the supervision of Doctor
Gloria DallAlba from the University of Queensland.

Signature: ..

Date: ..

166

Appendix 3

Printed on Official University of Queensland paper

PARTICIPANT INFORMATION SHEET

Project Title

Project Researcher

Identifying minimum standards in Airline pilot non-technical skills: A

phenomenological study.
Doctor of Education Thesis
Tim Mavin
Tel: +61 (0)7 XXXX XXXX
Fax: +61 (0)7 XXXX XXXX
Mobile: +61 (0) XXX XXX XXX
Email: timmavin@bigpond.com

My name is Timothy John Mavin and I am completing my Doctor of Education degree at the
University Of Queensland School Of Education. I am currently conducting research into the
identification of minimum standards in Airline Pilot non-technical skills. This research has
been approved through the University ethics committee.
By interviewing a number of experienced Check Captains, I plan to investigate their
experiences in assessment of pilots, especially in regard to their initial check as a Captain. I
believe the findings within this study will assist the airline industry to better understand the
standards that have been and are currently applied to pilots within the industry.
The interview will occur in a place that is suitable to yourself. During the interview, you will
be asked to describe some of your experiences as an experienced airline Check Captain. To
assist me in reviewing the information, I am requesting that I am able to audio record the
interviews. At a later date, I may ask you to clarify some items, so that your statements are not
incorrectly interpreted. This can be done at your convenience.
All information that I gain from this interview will, as per university guidelines, be kept in the
strictest confidence. Furthermore, should you wish to withdraw at any time during the research,
you may do so without prejudice. No names of pilots or airlines will be used within the
reporting of this research.
As mentioned, this research has been approved by the University of Queensland. If you would
prefer to talk to an independent person about your involvement, could I direct you to my
principal advisor Dr Gloria DallAlba on (07) 3365 6658 or the School of Education Ethics
Administrative Officer on (07) 3365 6502.
Thankyou for your cooperation

Tim Mavin

167

Appendix 4

Appendix 4 Interview schedule

Part A
Initial points to be made to Check Captain
1. Purpose of the research
2. As per the Consent Form, the interview and the data that is collected will be kept in
the strictest confidence.
3. At any time during the interview, you may withdraw for the interview without
prejudice.
------------------------------------------------------------------------------------------------------Part B :

Overview
Could you give me an example when you checked a pilot for initial command?
Could you think of another example of when a candidate failed/passed?
Could you give an example when the decision to pass or fail was difficult?
How do you feel when you have to fail a candidate?
What does the term minimum standard for initial command mean to you?

Simulator
Could you give me an example of when you checked a pilot for command in a simulator?
How do you see the difference between aircraft and simulator checks for initial commands?

Aircraft
Could you give me an example when you checked a pilot for command in an aircraft?
How do you see the difference between aircraft and simulator checks for initial commands?

Experience as a Captain
Have you changed the way you assess as you became more experienced?
Prompt:
Standards
Your approach
Second chance
Ease of failing someone
Have you changed the way you assess initial commands as you became more experienced?
What makes a good Check Captain?

168

Appendix 5

Appendix 5 Sample of CC3 interview and transformations

Meaning Units

Transformation 1
(CC3TR1)

TR2, in this case, is a

summary of TR1 (2, 3)

1st meaning unit

The interview for CC3 is split into the first meaning unit
below (2, 3, 4 & 5). From here, the interview was
transformed into two statements (2,3) under CC3TR1.

2.

R:
Can you just give us an example of, um, a person that
you have maybe done a check recently, or with regard to initial
command [promotion to airline captain]?

3.

CC3:
Yes, ah, in which area are you wanting me to go
down?

4.

R:
Um, simulators or on line [assessment whilst flying in
real aircraft], it doesnt matter because eventually we will
probably cover most areas over the next hour and a half or so.

5.

CC3:
Yeah. For most it is only really an extension of what
they have been doing over the last however many years while
they've been first officers, or whatever field, and, ah, the
difference being, they now have to manage the flight and it is
their total responsibility, rather than being, assisting the
responsible person in the past. (R: mm) And most people can
fly the aeroplane okay, and it is the command thinking and
command decisions, rather than the operation of the flowing,
ah, manner (R: mm) and being able to command people, that
makes the difference, and that is where people have the
difficulty if there is going to be a difficulty.

Transformation 2
(CC3TR2)

2.

3.

After being asked to give an

example of an initial command
upgrade assessment, CC3 gives a
broad view that being a captain is
an extension of the job they have
been doing as an FO. Especially in
regard to their flying skills usually
being up to standard. However, the
difficulty arises when candidates
are required to manage the flight
and become totally responsible for
the operation, rather than the
assisting role they played in the
past.
CC3 outlines that, if problems are
going to exist, they generally occur
in the area of command thinking,
decision making and commanding
people.

1) (2, 3)
The difficultly of transitioning
from the role of FO to captain
is generally related to the
candidate having to manage
the operation and its people,
make decisions, and take full
responsibility for its operation.

Transformation 3
(CC3TR3)

TR3 is developed
from a combination of
TR2 Transformations.
In this case, decision
making is made up of
TR2 11, 1, 6 & 30.

Decision Making
Candidates must be able to
obtain and prioritise relevant
and accurate information for
decisions to be made (11).
Difficulties
arise
for
candidates who are unable to
manage the operation (1).
Once decisions have been
made,
this
must
be
communicated to other crew.
It is expected that candidates
do not show signs of
hesitance,
changing
decisions or a reluctance in
making decisions (6, 30).

169

2nd meaning unit

Appendix 5
6.

R:
Yep. Can you give us a um have you done one
recently, a command upgrade?

4.

7.

CC3:
Ah yes, I am trying to think who and where and when.
Um, simulator I have done about three or four months ago, (R:
yep) but I have done a line check [assessment whilst flying in
real aircraft] on somebody (R: mm) more recently, and, ah, no
problems.

CC3 outlines that some upgrade

assessments that he has recently
conducted were assessed to be very
good.

5.

CC3 outlines that during the line

assessment, he sits in the FOs seat
to observe the candidate in a way
that enables him to take command
if required. If the candidate
performs satisfactorily in dealing
with the operation, a line FO is
allowed in the FOs seat, and CC3 is
then able to observe and assess all
areas and also how he operates the
aircraft with a line FO.

8.

R:
Okay. Ah with regards to the line check, how did it
go? It doesnt matter whether it went good or bad.

9.

CC3:
Yeah, they actually went very well. Once again, it's,
you just follow them around and you assist when you are the
First Officer on the first day, (R: yep) just so that if they are not
up to speed, you are there in the seat to take command.

10. R:

Yeah.

11. CC3:
And ah, you just monitor the tricky places and the
decisions that they make and how they handle things, how they
handle the little problems with the cabin. The cabin crew can
come up and say, We have got this problem with this
passenger (R: yep), or we are double booked [two passengers
being given the same seat at check in] or whatever, and how
they handle those situations. And then, ah, if you can see them
on the right track and they are not a threat to anybody, you get a
first officer in the seat the next day (R: yep) and I sit in the
jump seat [seat behind captain and FO] where I can observe in a
remote situation (R: yep) and observe the crew (R: mm)
situation between he and the new First Officer. And once again,
just to watch and assessing the performance (R: mm) in all
areas that is required for command.
12. R:

Yep, that is over how many days?

13. CC3:
Usually two (R: okay), but it can be three, but it is
usually two.

2) (5)
The final line assessment is
usually conducted with the
check captain acting as the
FO, and then the check captain
observing the operation with
the candidate acting as
captain, and a line FO acting
in their normal position.

Information
Candidates must be able to
obtain relevant and accurate
fact and information. With
this information, they must
be able to prioritise, so they
are
able
to
develop
situational awareness and
make effective decisions
(11, 12).
Situational Awareness
Candidates must be able to
maintain
situational
awareness at all times in the
operation. Those not able to
do so are unlikely to be
successful. It is extremely
important the candidate
obtains relevant information
so that they can develop an
accurate picture of the
operating environment in
order to safely operate the
aircraft (12, 29, 31). There
appears to be a link between
situational awareness and
decision making (31)

Management
Candidates must be able to
manage the operation so that
rushing does not occur (16)

170

Appendix 5

3rd meaning unit

14. R:
Okay. Have you done one in the past where a decision
to pass or fail has been a difficult, one where you have just had
to sit there and scratch your head just a little bit?
15. CC3:
Yes, oh yes, they come up, and ah, it is never easy (R:
mm), but a Check Captains position, although he is employed
by [current company], when he is in a checking role, he's got
his [countries regulator] hat on (R: mm) and he is a direct
employee if you like of [countries regulator] and they cant
provide um check people to do unlimited checks in [this
country], so that is why they designate (R: mm) airline staff to
be (R: yep) in that role. So whether it is going to reflect on the
airlines performance, that is totally immaterial. You are just
looking at it through [countries regulator] eyes, and you've been
entrusted with this job (R: mm), and there is a certain standard
that [countries regulator] lay down and most airlines have a
higher standard again, and most pilots have a higher standard
again on top of that. (R: yep) They have their own personal
standard. (R: Yeah)
16. and ah if the guy doesnt ah make at least the airline standard,
well then you start to scratch your head and think, well, I am
not sure about this, and you can usually work it out (R: yep),
and sometimes it is to the benefit of the guy anyway, to go back
and just do a little bit more training. (R: mm) It is like building
a house and putting a good foundation under it and you end up
with a good product. If you skimp a little, it is not quite up to
speed and then you ah, build a house on top if it and it never
gets there (R: yeah okay) um, the bottom line I suppose with
all check captains is in those head scratching situations, (R:
yep) how would I feel on a dark and stormy night with my wife
and kids coming back with this guy. (R: Yep), and if you know,
well yeah I would be comfortable with that, (R: mm) you know
what the answer is, (R: yep) but if you think, oh no I'd be a bit
nervous about that, it is a different answer.
17. R:
Yeah. So can you think of one in the past where it was,
the decision was a little bit difficult?
18. CC3:
past.

6.

CC3 outlines that during

assessments, he is acting as a
delegate of his countries regulatory
authority. He declares that this
delegation requires the assessment
to be conducted in an impartial way
and not be influenced by the
particular airline.

7.

CC3 advises the regulatory

authority has a required standard,
the airline generally has a higher
standard and the individual pilot has
a higher expectation of themselves
above that.

8.

CC3 advises that candidates must at

least be to the standard of the
airline. If there is doubt around this,
it would be to the benefit of the
candidate to be given more training.

9.

To better assist the CC3 deciding if

he would allow a candidate to pass,
he hypothesises about whether he
would allow them to be a captain on
an aircraft with his family on board
the aircraft.

10. CC3 discusses that he has not

recently had a marginal assessment,
though has had them.

Ah not terribly difficult, no (R: okay) not in the recent

11. CC3 refers to fellow check captain
who have actually passed

3) (6,7)
The check must remember that
whilst they are assessing
candidates, they are doing this
job as a countries delegate and
not
as
a
company
representative.

Communication
It is important that a
candidate is able to share
and improve their situational
awareness, the situation
awareness of crew and
effective
and
decision
making process through use
of communication (10).
Flying Skills
Candidates must be able to
manipulate the aircraft to a
reasonable standard (9).

4) (8,9)
The candidate must at least
meet the minimum standard of
the airline. This standard is
usually higher than that of the
regulatory authority but below
that of the individual check
captain.
This
standard
sometimes relates to how
comfortable the check captain
would be to allow his family
to fly with this candidate.

5) 11)
The check captain outlines the
importance of being certain
about allowing candidates to
pass. He is aware of other
check captains who have

Assessment
The simulator assessment is
a more stressful assessment
(18). Check captains must
obtain sufficient evidence to
correctly make decisions on
a candidates ability by
obtaining as much evidence
as possible and if there is not
enough evidence, or doubt
exist, the candidate must
require further assessment
(5, 8, 15, 24). It is important
for candidates to be able to
accurately self assess their
own performance (27, 28)
One of the difficulties new
Check Captains have is
appreciating the various
ways individuals can operate
aircraft
operate
within
company
operating
standards (21).

171

Appendix 5
19. R:

Okay. How about a little bit further back?

20. CC3:

Yes. (R: mm) Everybody has had them.

21. R:
Yeah, absolutely, which is the ones that I am trying to
get out.
22. CC3:
It has never happened to me, but it has happened to
others that I know and have passed somebody (R: mm) and
who have rung up the flight department the following day and
said, I made a mistake, I shouldnt have passed him. (R: Yep)
and it is too late, (R: mm) you cant turnaround (R: yeah) and
pull the rug out after you have signed that bit of paper (R:
Yeah) so it is one of those areas that you must be positive (R:
Yeah). It is such a crucial area and you are putting somebody in
charge of an aircraft which can have up to 500 people (R: yep)
in it. (R: mm). You dont want to be biting your fingernails
every time that guy signs on [starts work]. (R: yeah). So if
anything you err on the side of if it's a line ball, will I or won't
I, (R: yep) no I wont! (R: Yeah). Give him another 20 hours
or whatever it is (R: yep) and just make sure.
23. R:
Can we maybe talk about one that you've um, you
were erring then, that you have said, No I wasnt happy, I
might just give that person an extra bit of training. Can you
think of one that might have occurred over the last recent times
where you think the most recent one, as I said, it might be six
months, 12 months or a couple of years?
24. CC3:
.Yesum the guy just hadnt really taken the next
step (R: mm) from being a First Officer (R: mm) to
commanding the situation. He was tending to be a little bit
uncertain of some of his decisions, (R: mm) but what he
needed waswas that it was a good decision or whatever,
rather than taking command of it and saying, Right, grab it by
the throat, and (R: yeah okay) whether I like or not this is
what we're going to do.

candidates to later regret their

decision. CC3 describes the
responsibility of getting the
decision correct, and believes if
there is any doubt as to the standard
of the candidate, then you must
allow more time to train or assess
so as to certain of the decision.

regretted passing individual

candidates.

Standard
Operating
Procedures
Candidates
need
to
understand how to operate to
company standard operating
procedures and recognise
when they can deviate from
them (23).
Airmanship
Candidates must possess
sound airmanship (32).

12. CC3 said a candidate was

unsuccessful due to being uncertain
of decisions they made. He
indicates that this signified a
candidate who had not made the
step from FO to captain.
13. CC3 describes that, with some
candidates, you are able to pick
very early if they are going to
struggle with the operation. Others
may make decisions on the last day
of the assessment that concern the
check captain.

6) (12)
Candidates must give not give
the impression that their
decision making is uncertain
and hesitant.
7) (13)
In some cases, it is easy to
predict that some candidates
will have difficulty with
assessment, whilst other may
perform well and make
decisions that place their
judgment into question.

25. R:
Okay. Can I get you on that one then? Can I, this is
where I will start going back. When you signed on and you
might think that I cant remember some of this or whatever and

172

Appendix 5
that is fine, but can you think about when you began the check
with this guy and walk me through the process of what you
were seeing um that made you start to think that maybe this is
going to be a hard decision for yourself to make, therefore you
have erred on the cautious side?
26. CC3:
Probably it is not always a sign and it might be on the
last day (R: okay) of the check and suddenly it happens in the
air, this bloke's not, that's not right, that shouldnt that is a
wrong decision and this could really affect (R: mm) the safety
of this operation. But others, well at sign on you can pick,
yeah this bloke's is not too sure of his fuel (R: yeah)
27. and nerves play a big part too with these guys. (R: mm hmm).
So you've got to try and sort out the nerves side of things a little
bit and you cant just say, Well I will just put that all down to
nerves, you cant do that (R: yeah) and you have got to play it
as you see it. (R: Yep). But I think that most people under
check, whether it be a simulator or a line check, (R: mm)
probably only operate to about 80% of their normal capacity or
ability, due to nerves. (R: Yep). They are fingers and thumbs
because they are observed, and things like that. But you cant
make allowances for that, even if you have flown with the guy
as First Officer, and you flew with him last week and you know
him very well and you know how he flies, if he gives a bad
check, that is when you go to another level. (R: yeah) You
cant say, Well I know you can do better than that, so I will
pass you. (R: Yeah). Today is what you pass him on.
28. But getting back to your original question, (R: yep) um there
are so many things that can be affected that they might totally
misread a forecast and (R: mm) therefore not really put enough
fuel (R: mm) in the aeroplane to cover the requirements to
makes it legal. (R: Yeah). Umhe may ahinadvertently bust
an altitude somewhere over by 2 or 300 feet and it is just not
good enough, (R: yeah) you have got to do it right. Or it might
be a um, a decision to divert when he shouldnt have done that
and the weather is fine, but for some reason he's just diverted
and said well I've erred on the side of safety, so therefore it is
okay. (R: yeah) Well it is not okay, if everybody did that,
aeroplanes would be at different airports where they're
supposed to be every night of the week. There will be

14. CC3 believes check captains should

not make allowances for nerves, or
for previous experiences they have
had with the candidate. He declares
that an assessment must be what is
observed rather that making
interpretation on what their
performance could or should be
like.

8) (14,15)
Assessment of a pilots
performance must be made at
that time, and should not allow
for areas of influence such as
nerves
or
previous
performances. Additionally,
there are many areas that can
be assessed.

15. CC3 outlines that there are many

areas that can go wrong. These may
include misreading information, or
taking a too cautious approach to
the operation that impacts the
commercial operation of the airline.

173

Appendix 5
thousands of people getting bussed all over the country. (R:
Yeah). So they have got to put in a fair effort and a fair (R:
sure) sort of a job.
29. R:
Yep. So if I can get back to this person before where
you said his decision making, was it decision making that you
were a bit...
30. CC3:
It could be, (R: yeah) but it could be heaps of things.
31. R:
So with regard to um, was there anything, because
what I am eventually after is concrete examples, and was there
anything, was there maybe a couple of examples of things that
he did that you can explain to me that made you realise that, no
this is now pushing a little bit down on the ?

16. CC3 describes a candidate in the

simulator who was unable to
position the aircraft in a way that
would facilitate a safe landing.

10) (18, 20)

Even if a candidate has made a
safe decision, it is important
that they communicate the
decision,
and
important
information, to other crew
members.

32. CC3:
Well there was, we had a cloud base of 1,900 feet and
at a sea level airport (R: mm) and he, ah, made two attempts to
get in (R: mm) on an instrument approach (R: yep) and didnt
and then just decided he would divert, just because he couldn't,
he was messing up the approaches. The approaches were safe
and there was no risk to him coming into contact with the
ground, (R: yeah) but the fact that he couldnt get the
aeroplane into position to land, (R: okay) he just decided okay
he would decide to divert. (R: Okay). In that case there was
no...
33. R:

Was this the sim?

34. CC3:

yes that was that a simulator? Yeah!

35. R:
It was sim, yep.
36. CC3:
In that case, there was no reference to how much fuel
he had left. He probably knew how much fuel he had left but he
didnt mention it to the First Officer, I have got x amount of
fuel left and we are going to burn this to get to where we are
going, (R: yep) so therefore we are okay, and that was
never mentioned. So we diverted and ah landed, and landed
with the required reserves, (R: yep) but there had been no
mention of it throughout the whole deal. And ah when we
bought it to his attention, we ran it again in fact, and ah, he still

9) (16, 17)
Candidates must be able to
manipulate the aircraft to a
reasonable standard.

17. CC3 describes a candidate in the

simulator who was unable to
position the aircraft in a way that
would facilitate a safe landing.
18. CC3 outlines that, after the
candidate was unable to land the
aircraft, they decided to divert to
another airport. CC3s concern was

174

Appendix 5
couldnt do it. (R: Okay). So when this was bought up in the
debrief, his argument was very aggressively, well it was a safe
operation. You know, I went to work and it was a safe
operation and (R: mm) that is all I have got to do. (R: Okay).
And that is why I made the point, well if everyone did what you
did, then when it's an airport that you can easily get into, (R:
yep) the company doesnt want its aeroplanes stuck in the
wrong places all over the [country], because people just cant
get in at the end of the day.
37. R:

that he did not effectively

communicate the diversion facts to
the FO. CC3 outlines that the
candidate argued the operation was
safe, although CC3 believed
otherwise.

Yeah, yeah.

38. CC3:
And then the following day the aeroplane is in the
wrong place for the early flights and the crew as they are
coming up for those aeroplanes, there is no aeroplane there and
they are all somewhere else (R: yep) and you know, I said,
how can the airline possibly survive with people like that.
39. R:
But was it his manipulation, he just couldnt get the
aeroplane in the right position or?
40. CC3:
Yes pretty much, um. Yeah he was getting too close to
the field and then trying to do a circuit over the top of the place,
and losing it, and then he would come around the corner and
you know, be way off the centre line and just couldnt get back.
41. R:
Okay. Was that like, where was it, [capital city airport]
or something?
42. CC3:
It was in [major city airport].
43. R:
The [major city airport], okay. Because that is exactly
what I am after. I'm after these specific examples where um
rather than a generalisation, I would love, what the researcher is
about, if I can get these specific examples of what guys have
done that you said, that's what I didnt like or that was good or
that is exactly what I am after, if that makes any sense.
44. CC3:
With this guy, it was easy to know exactly how he is.
45. R:
(Laughs). And as I said, it doesnt matter who the
person was and I probably didnt know them anyway so.

19. The researcher reconfirming his

request for specific examples of
assessment.

20. Although the candidates

manipulation was unable to position
the aircraft in a way that would
facilitate a safe landing, CC3

175

Appendix 5

46. CC3:

I think you would.

describes decision making as the

main reason for this candidates
failure.

47. R:
(laugh) Um can you think of anything else with regard
to where I am really looking at those things where after four
hours in a sim or two days in a line check, you might say,
Look it was three or four things, but this is what I saw that
made me think no.
48. CC3:
It really all boils down to the decision making. (R:
Okay). As I said, most people can fly the aeroplane and this guy
we just spoke of, okay it was more manipulation, that he
couldnt get it in. (R: Yeah). But that was a one-off situation,
and the rest of his manipulation was okay and it was just that
one time, he just kept getting the aeroplane into a position
where he couldnt land and (R: yep) decided he would divert.
(R: Yep).
49. And another one ah a guy had a fire in the rear galley (R: okay)
and the girls were giving him ah prompts with just fired the
fourth bottle into this thing and it was still going. (R: yep). That
should be a real flag to the guy (R: mm) to get this thing on the
ground pronto because we have got an uncontrollable fire. (R:
Yep). But he allowed air traffic control to push him all around
the place and he was even going a lot slower than he should
have been anyway before the approach, and ah, with no
urgency and no forward planning that we are going to stop on
the runway or are we going to have the firies [airport fire
service] standing by on a certain point on the runway are we
going to stop right beside them so that they can get on board
and get this thing out. (R: Yep).
50. And that comes back to just, you know, he could fly the
aeroplane, (R: yep) but he just wasnt thinking like a Captain
and the priorities were all wrong and thats where um the
decision was really fairly easy. (R: mm). It is very rarely one
thing. (R: Yeah).
51. It's, you know - for instance on another occasion we had a guy
we had a thunderstorm on track, (R: simulator?) simulator (R:
yep) and he was going around the thunderstorm, (R: yep) but
he kept nibbling away at, to get back on track and he ended up
going right through the top of the thing. (R: mm). Well you're

21. Even though a candidate had been

given important information from
the cabin crew about the urgency of
a fire in the rear of the cabin, CC3
describes a candidate who was not
prioritising these facts and making
appropriate decisions with this now
known information.

22. CC3 outlines the assessment was

deemed below standard due to the
candidate not thinking like a captain
in the way they were prioritising
information.
23. CC3 outlines a candidate who was
not obtaining relevant information
about thunderstorms in the area.
CC3 describes the candidate
diverting to an aerodrome that was

11) (21,22,23)
Candidates must be able to
obtain relevant and accurate
fact and information. They
must be then able to prioritise
this information so they are
able to make effective
decisions.
12) (24)
Candidates must be able to
obtain relevant and accurate
fact and information so that
they are able to develop a
mental
picture
of
the
environment in order to safely
operate the aircraft.

176

Appendix 5
prepared to accept that and say that we will have a good talk
about this afterwards, but if that is the only thing he does, we
will just talk about it. It will still be okay. [Capital city airport
#1] weather was um okay using landing into the south and it
was pretty much straight in for him. (R: Yeah). And ah, in the
ah, information aerodrome information service, they said there
was thunderstorms to the south. Well that is spooky (R: yep)
and we are 125 miles from [capital city airport #1] and we are
250 miles from [capital city airport #2] at this stage; nautical
miles. So because of thunderstorms in the south, he elected to
turn around and go back to [capital city airport #2]. (R: Okay).
So in the process, he went clean through the top of this
thunderstorm again.
52. R:

So he did it twice?

53. CC3:

Yeah.

54. R:

Okay.

some distance from the original

destination, due to storms.
However, CC3 concern was the
candidate did not obtain relevant
information about these storms
prior to the diversion.
24. CC3 was also concerned that the
candidate was not aware of the
outside environment in the way that
they flew the aircraft twice through
thunderstorms. He put this down to
poor situational awareness.

55. CC3:
And you know, that was really enough and you say, oh
this is just, This is not on.
56. R:

What did you put that down to?

57. CC3:

You cant go through the thunderstorm twice.

58. R:
Why would he? If you could put it into words, what
was worrying him? Was there anything that you ...
59. CC3:
The thunderstorms, they could have been down over
[geographical area south of capital city airport #1] and it would
have been just a thing they could see from the tower down that
way. And ah there was no reason for him to divert back to
[capital city airport #3] anyway.
60. R:

Yeah okay.

61. CC3:
But by going back through the top of thunderstorm,
that really put a question mark over him, but then the wheels

177

Appendix 5
really feel off, I dont know whether he lost confidence after
that, but it just um fell apart after that and he got to the stage
where going to [capital city airport], it was just high and hot
and couldnt do it.
62. R:
So he had gone, so was that [capital city airport #1],
[capital city airport #2]?
63. CC3:
No, he was going the original flight was from
[capital city airport #3] to [capital city airport #1].
64. R:
Okay, [capital city airport #3] to [capital city airport
#1] yep.
65. CC3:

And ah, that was with a thunderstorm on track.

66. R:

Okay, yeah.

67. CC3:
And ah yeah, so those are the sort of things and they
are fairly clean cut easy decisions, so...
68. R:
So he was going [capital city airport #3] to [capital
city airport #1] and the reason he didnt go to [capital city
airport #2] was because of storms?
69. CC3:

There were thunderstorms to the south was all it was.

70. R:

Okay.

71. CC3:
And expecting into you know, he was only 20
minutes away (R: okay) but he turned around and went back
(R: okay) and in the process went clean through this massive
storm.
72. R:
Okay, so he wasnt aware then what was outside the
aeroplane?
73. CC3:

Yeah.

74. R:

Yeah, okay. What would you, how would you define

178

Appendix 5
that with regard to...?
75. CC3:

Well um lack of situational awareness.

76. R:
Mm hmm.Okay.
77. CC3:
Um(8 sec)line checks, you occasionally get
people um.it is usually lack of familiarity (R: mm) that will
bring people undone on a line check, some people, not others.
(R: yep) But ah, I wasnt associated with it, but the guys told
me all about him in [regional city airport], on his line check and
that he hadnt been yet (R: yep) and it was at night and he had
messed it up. And when you take the nerves into consideration
(R: yeah) and hadnt been there before and you know,
[regional city airport] is a bit of a black hole (R: Yes it
is)..(R: mm) so he probably was ah a bit unlucky. (R: Yeah).
A lot of the other guys just get you know, a couple of [capital
city airport] and maybe [capital city airport] and back or (R:
yeah) you know [major city airport] or something like that,
where they've been fairly familiar so they know where it is. (R:
yep) And had he got one of those he might have been okay
from there. (R: Yeah). He just happened to get [regional
airport] and (R: and not ...) he hadnt been there before and it
was a dark night and whoever it was that checked him wasnt
happy.
78. R:
Okay good. Can I um , we will get back to more of
those later because I find that sometimes if you just leave them
and then come back if thats okay. Um with regard to um some
of the command upgrades in the sims that you have done as
well, um can you really give us a more of a concrete example
on those ones as well? Because you've already given a couple,
but I was just wondering if there is anything that you can
specifically think about with regard to the sim that you might
have done, where in regard to it was a marginal, marginal pass.
It doesnt matter if you passed them or failed them, (CC3: um)
but specifically in the simulator? ......
79. CC3:
It's usually, if it's marginal it's usually, yeah there is
such a thing as a marginal pass.
80. R:

Yep.

25. CC3 outlines that familiarity with

individual airports is important
factor in performance. He suggested
that a check captain who assessed a
candidate operating into an
unfamiliar airport at night, and with
the expected nerves, may have been
a little unlucky.

13) (25)
Familiarity with airports aids candidates must being able to
obtain relevant and accurate
fact and information.

26. Although CC3 does not expand, he

believes there is a difference
between a marginal pass and a
marginal fail.

14) (26,29)
A marginal pass may occur
when very few events that did
not effect safety and can be
debriefed. A marginal fail may
occur when there were a
number of areas of concern
that may not be able to be
debriefed, or one that effected
safety did occur.

27. CC3 outlines that if a he was

closely considering if he should
pass a candidate or not, he would
probably require more training.

15) (26,27)
If any doubt exists as to the
standard of a candidate, it is
recommended
that
more
training being given.

179

Appendix 5

81. CC3:
fail.

But a marginal pass is quite a step up from a marginal

82. R:

Yepokay.

16) (28)
It is important that candidates
manage the operation so that
rushing does not occur.

83. CC3:
You know if it was really border line and you couldnt
make up your mind which way to go, you have the concerns,
you'll weigh it and say no, we will give him some more time.
84. R:
Okay, yep.
85. CC3:
Um(11 sec)one of the things I think in a
simulator situation where we are simulating failures and
emergency procedures and whatever, people are out of their
comfort zone (R: yep) but I dont care who you are, Check
Captains, (R: mm) everybody is out of their comfort zone
whenever there is anything abnormal going on. Because for six
months of the year we dont see it (R: mm), and the planes
have been reliable (R: absolutely) and nothing ever
happensthankfully (R: yep). But in a simulator, guys will
tend to rush. Lets get this off our plate and get back into our
comfort zone (R: mmmm) and that is where they tend to
make mistakes. Where they should enter a holding pattern, give
themselves a bit of room, no, no hold up we can make this
straight in and we wont have to do a hold, we'll get - and they
start their approach, (R: yep) 500 feet after they have started
the approach, Oh not we're running out of time here, oh lets
go back to the holding pattern.
86. R:

Yep

28. CC3 outlines a tendency for all

pilots to rush during simulated
exercises in the simulator. He
understands why they rush, though
he suggests it is a very bad sign and
accordingly is unable to make
allowances for this rushing
phenomenon in the assessment
process.
29. CC3 differentiates a marginal pass
being where a candidate does
something wrong but is able to be
debriefed on the exercise as long as
the indiscretion did not threaten
safety. Whereas a marginal fail may
occur where the candidate actually
did more than one ordinary
performance areas or they
threatened safety.

87. CC3:
It's a very, it is a very bad sign. I know why they are
doing it, but I cant tell you that I'm going to make an
allowance and say, well that is okay.
88. R:

Yeah okay.

89. CC3:
But you also might get a guy that that is all he does (R:
mmm) and that is the one misdemeanour in the whole day that
he did.

180

Appendix 5

90. R:

Yep.

91. CC3:
So it is not actually a marginal pass or fail, you have a
good talk to him about it and say, Listen, dont you rush, use
sit on those hands and have a good look (R: yep) and if you
need a holding pattern, you go into a holding pattern, (R: yep)
But..Dont..Rush! By the time you're in a holding pattern
you are going to be ready to land and you are going to be in a
landing configuration. And so that would be I guess a marginal
pass, because had he done one or two other things that were a
bit ordinary (R: yep) you would say, Nosorry.
92. R:

Yeah.

93. CC3:
But in this case, you know, there was no risk involved
(R: yeah), he wasnt threatening safety (R: yep), it was just a
poor decision.
94. R:

Yep

95. CC3:

So we will let you go.

This interview continued to 329

181

Appendix 6

Appendix 6 Combined final transformation (TR3) of all five check captains

CC1TR3

CC2TR3

CC3TR3

CC4TR3

CC5TR3

Information
The candidate is expected to
obtain accurate and relevant
information to develop situational
awareness (18).

Information
Candidates are required to be
able to obtain information from
various sources so that they are
able to carry out sound decision
making, and maintain situational
awareness (8, 21).

Information
Candidates must be able to obtain
relevant and accurate fact and
information. They must be able to
prioritise this information so they
are able to develop situational
awareness and make effective
decisions (11, 12).

Information
Candidates ought to act in a way
that enables them to obtain as
much information as they can (9).

Information
Candidates must be able to source
information to develop situational
awareness, and for decision
making (13).

Situational Awareness
The check captain places great
importance
on
candidates
maintaining situational awareness
(12, 18, 53, 54). It appears that
candidates
must
maintain
situational awareness at all times,
whilst demonstrating how they
share and improve their own
situational
awareness
by
communication with the other
crew members (9, 11, 18, 22, 31).

Situational Awareness
Situational awareness is difficult
to assess (2, 48). It is extremely
important,
particularly
as
situations become more complex.
Candidates
must
maintain
situational awareness throughout
the entire operation (3, 16, 18,
21, 39). Candidates who are able
to maintain their situational
awareness throughout the entire
operation are less likely to fail
(28).

Situational Awareness
Candidates must be able to
maintain situational awareness at
all times in the operation. Those
not able to do so are unlikely to be
successful. It is extremely
important the candidate obtains
relevant information so that they
can develop an accurate picture of
the operating environment to
safely operate the aircraft (12, 29,
31). There appears to be a link
between situational awareness and
decision making (31)

Situational Awareness
Candidates must be able to
prioritise the operation so that
they are able to operate the
aircraft as accurately as possible,
and not lead to a loss of situational
awareness (1). Candidates must be
able
to
source
accurate
information so they are able to
develop accurate models in order
to operate the aircraft accurately
(21, 26).

Situational Awareness
Situational awareness is the
understanding of where the
aircraft is in space, and is an
extremely important area that is
assessed in command upgrades (2,
7, 9). Candidates must be able to
source information from the
environment to develop their
situation awareness, and be able to
share their situation awareness
with other crew members (13).
Candidates must maintain
situational awareness, though
failure to maintain situational
awareness may be seen in other
areas like flying skills (19).

182

Appendix 6

Decision making
Candidates are expected to be
able to make sound decisions
under pressure (5). These
decisions are expected to take
risk into account (32).

Management/Leadership
Candidates should act in a way
that is conducive to other crew
members being able to question
them, and be open to suggestions
(32, 53). Candidates should also
be able to manage any situation
that arises in the flight deck (55).

Responsibility/leadership
&
Management
It is expected that candidates are
able to manage the operation.
However some level of leniency
may be given if complex
scenarios are encountered (40).
The candidate must be able to
manage the operation in a way
that enables them to maintain
their situational awareness at all
time (24, 26). The candidate is
expected
to
display
characteristics
that
show
subordinates that they are in
charge of the operation (49),
which
includes
motivating
subordinates without the use of
their rank (43). Candidates are
expected to be able to prioritise
operational tasks (23).

Decision Making
Candidates must be able to obtain
and prioritise relevant and
accurate information for decisions
to be made (11). Difficulties arise
for candidates who are unable to
manage the operation (1). Once
decisions have been made, this
must be communicated to other
crew. It is expected that
candidates do not show signs of
hesitance, changing decisions or a
reluctance in make decisions (6,
30).

Management
Candidates must be able to
manage the operation so that
rushing does not occur (16)

Decision Making
Candidates must be able to source
relevant information that is
discussed with other crew so that
sound decisions are made (13, 21).
Once this process is complete, the
complexity reduces (21).

Management
Candidates are required to coordinate crew activities (29).
It is important for candidates to be
able to identify the level of
importance of tasks, and to
appropriately prioritise operating
them (23, 25).

Management
Candidates must be aware of the
required tasks, and act accordingly
in order to have them carried out
by themselves or other crew
members in a way that enables
them to maintain situational
awareness (8, 12, 19). Candidates
who do not effectively manage the
operation impede other areas such
as situational awareness, and
flying the aircraft (19, 20).

183

Appendix 6
Communication
Candidates are required to be able
to clearly share information in
relation to situational awareness
and decision making (29, 31, 54,
56).

Communication
It is important that a candidate is
able to share and improve their
situational
awareness,
the
situation awareness of crew, and
their effective decision making
process
through
use
of
communication (10).

Flying skills
The check captain does not outline
in detail the ability of the
candidate to fly the aircraft, as it is
assumed to be second nature to
them (36).

Flying manipulation
Candidates must fly the aircraft
within the tolerances that are set
down by the regulatory authority
(2).
Small
digressions
in
manipulation are allowed if the
candidate is able to maintain
situational awareness and the
digression is managed (44).

Standard Operating Procedures

Standard operating procedures
play important but not critical
requirements for assessment (6).

Standard operating procedures

& basic knowledge
Candidates are expected to
operate aircraft within the bounds
of company standard operating
procedures (9).

Flying Skills
Candidates must be able to
manipulate the aircraft to a
reasonable standard (9).

Standard Operating Procedures

Candidates need to understand
how to operate to company
standard operating procedures and
recognise when they can deviate
from them (23).

Communication
Candidates should be able to
effectively give and receive
information between other crew
members (12, 17, 21). Effective
communication aids candidates in
sharing their SA and intended
course of action via the decision
making processes (12, 13, 21).

Technical Skills
Manipulative standards are those
laid down by the regulatory
authority (5).

Flying Skills
Candidates must be able to
smoothly manipulate the aircraft
within company and regulatory
limits. The inability to do this may
be a manifestation of management
problems, or lower situational
awareness (11, 19, 22).

Standard Operating Procedures

It is important for candidates to be
able to understand and follow
standard operating procedures.

184

Appendix 6
Individual
Candidates who are undergoing
command upgrade training on a
new aircraft type appear to find
the task more difficult (1, 27, 42).

Self Assessment
It is important that candidates are
also able to make self assessment
judgements (13, 15, 27, 37).

Airmanship
Candidates must possess sound
airmanship (32).

Knowledge
Candidates are required to have
some level of depth to the
understanding
of
technical
knowledge (5).

Risk
The ability to differentiate the risk
associate with differing courses of
action is important for candidates.

Non-technical skills
Are general skills that would
relate to those in the community
such as communication and body
language. It also includes morals
and ethics (6, 7).

Cooperation
Candidates are expected to
develop some level of cooperation
with the FO (7).

The End

185