Safety Assessment Methods

DIVISION OF AIRPORTS AND AIR TRAFFIC SAFETY
FACULTY OF TRANSPORT AND TRAFFIC ENGINEERING

UNIVERSITY OF BE LGRADE
Safety Assessment
Methodologies
Fedja Netjasov
University of Belgrade
Faculty of Transport and Traffic Engineering
Air Transport Department
Introduction
Introduction (1)
Among the class of complex and safety critical industries, air

traffic is an interesting example that poses exceptional challenges to
advanced design.
By its very nature, each aircraft has its own crew, and each crew is
communicating with several human operators in different air traffic
management (ATM) and airline operational control (AOC) centres on
the ground in order to timely receive instructions critical to a safe
flight.
In addition, from an organisational perspective, air traffic involves

interactions between many stake holders: pilots, air traffic
controllers, airline operation centres, airport authorities, government
regulators and the public travelling.
D IVISION OF AIRPORTS
SA
AND AIR TRAFFIC SAFETY
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
U NI VERS ITY OF BELGR ADE
Introduction (2)
Among the safety critical industries, air traffic stands out regarding
the many distributed levels of interactions in control and decision
making.
The implication is that safety of air traffic is the result of interactions

between multiple human operators, procedures (including spacing
and separation criteria), and technical systems (hardware and
software) all of which are highly distributed.
Since safety depends crucially on the interactions between the

various elements of a system, providing safety is more than
making sure that each of these elements function properly.
It is imperative to understand the safety impact of these

interactions, particularly in relation to non-nominal situations.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
Introduction (3)
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
Methods/Models of the Risk and

Safety Assessment (1)
Causal methods/models for risk and safety assessment of aircraft

and ATC/ATM operations. This category deals with failures of
particular technical systems and components resulting in the aircraft
crash. The failures can be due to many interrelated causes and
happen either in the aircraft or at ATC/ATM;
Collision risk methods/models are dealing with assessment of the

risk of aircraft collision while airborne and/or on the ground due to
deterioration of ATC/ATM separation rules;
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
Methods/Models of the Risk and

Safety Assessment (2)
Human factor error methods/models. This category deals with

risk and safety assessment of air traffic incidents and accidents due
to human error, and
Third-party risk methods/models. This category of

methods/models considers the risk assessment for people on the
ground, who might be affected by the aircraft crash.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
Causal methods/models for the

risk and safety assessment (1)
Causal methods/models of assessment of risk and safety of aircraft

and ATM/ATC operations establish the theoretical framework of
causes that might lead to aircraft accidents.
These methods/models can be qualitative or quantitative.

The former provide a diagrammatic or hierarchical description of the

factors that might cause accidents. They are useful for improving
understanding of causes of accidents and proposing preventive
interventions.
The later estimate the probability of occurrence of each cause and

hence estimate the risk of accident. They might be restricted to pure
statistical analysis based on the available data or combine these data
with expert judgment on the accident causes.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
Causal methods/models for the

risk and safety assessment (2)
In addition, they can estimate the relative benefits of different

interventions aiming at preventing accidents in the future.
Some of the methods/models are as follows:

Fault Tree Analysis (FTA);

Common Cause Analysis (CCA);
Event Tree Analyses (ETA);
TOPAZ accident risk assessment methodology;
Bayesian Belief Networks (BBN); etc.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
Collision risk methods/models (1)

One of the principal matters of concern in the daily operation of civil

aviation is preventing conflicts between aircraft either while airborne
or on the ground, which might escalate to collision.
Although aircraft collisions have actually been very rare events

contributing to a very small proportion of the total fatalities, they
have always caused relatively strong impact mainly due to relatively
large number of fatalities per single event and complete destruction
of the aircraft involved.
In general, separating aircraft using space and time separation

standards (minima) has prevented conflicts and collisions.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
10
Collision risk methods/models (2)

However, due to reduction of this separation in order to increase

airspace capacity and thus cope with growing air transport demand,
assessment of the risk of conflicts and collisions under such
conditions has been investigated using several important
methods/models as follows:

The Reich-Marks model;

The Machol-Reich model;
The intersection models;
The geometric conflict models;
Generalized Reich model; etc.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
11
Human factor error

methods/models (1)
Investigation of causes of particular air traffic accidents has

identified human error as one of the most frequent causes.
Human error is considered as an incorrect execution of a particular

task, which as an event, triggers a series of consecutive errors in
execution of other tasks, finally resulting in serious consequences
an aircraft accident crash.
Therefore, monitoring and modeling of human errors in the aircraft

and ATC/ATM operations aiming at discovering and preventing
them have always been high on the research agenda of both
academics and practitioners dealing with civil aviation.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
12
Human factor error

methods/models (2)
Consequently, many methods for detection and prevention of

human errors have been developed; some of them are:

HAZOP (Hazard and Operability) method;

HEART (Human Error Assessment and Reduction Techniques);
TRACER-Lite (Technique for the Retrospective Analysis of Cognitive
Errors);
HERA (Human Error in ATM);
HFACS (Human Factor Analysis and Classification System); etc.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
13
Third-party risk methods/models

Third-party risk implies risk if an individual on the ground to be killed

by crashing aircraft. In such a case, the accident is called a
groundling accident or groundling crash and the fatality a
groundling fatality.
Since most air traffic accidents happen around airports, the concept
and assessment of third-party risk has been mainly focused on
areas around airports.
Modeling of third-party risk has shown promise in resolving these

problems including setting up thresholds for acceptable risk around
airports.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
14
Design
approaches
15
Design approaches (1)

Traditional ATM design approaches tend to be bottom-up, that is

starting from developing concept elements aimed at increasing
capacity, and next to extend the design with safety features.
The advantage of the traditional approach is that advanced design

developments can be organised around the clusters of individual
elements, i.e., the communication cluster, the navigation cluster,
the surveillance cluster, the automation tools cluster, the
controllers/pilots and their human machine interfaces (HMIs), the
advanced procedures, etc.
The disadvantage of this traditional approach is that it fails to fully

address the impact of interactions between controllers, pilots and
ATM systems on safety.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
16

A goal oriented approach would be to design ATM such that safety

has been built in at the capacity-level required. From this
perspective, safety assessment forms a primary source of feedback
in the development of advanced ATM designs.
An early guidance of ATM design development on safety grounds

can potentially avoid a costly redevelopment program, or an
implementation program that turns out to be ineffective.
Although understanding this idea is principally not very difficult, it

can be brought into practice only when an ATM safety assessment
approach is available that provides appropriate feedback to the ATM
designers from an early stage of the concept development.
This feedback should provide information on which safety-capacity

issues are the main contributor to un-safety.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
17
A safety risk assessment methodology which provides safety risk

feedback to advanced air traffic operation design.
design
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
18
Generic
Safety Assessment
Process
19
Generic Safety Assessment Process (1)

Safety assessment is the process through which it is assessed

whether traffic increase in an existing operation or new proposed
changes do not sacrifice safety and preferably make things better.
This means that all possible impacts of a new operation or system

should be assessed, and their combined safety effects determined.
A safety assessment considers the proposed operation or system

definition (often called the Operational Concept), and communicates
these results with concept designers which could impact matters, for
the better and/or for worse, with respect to safety.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
20

Safety Assessment of an air traffic operation can be seen as a

seven-stage process, with two feedback loops:
The first refers to Iteration, meaning that safety assessment is usually

iterative in nature and safety assessments themselves are not always
once-through processes.
The second feedback loop is safety communication and feedback

leading to organisational learning. This communication should be part of
all other stages.
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
21
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
22
Generic steps of a safety

assessment
1. Scoping the assessment
2. Learning the nominal system
3. Identifying hazards
ITERATION
4. Combining hazards
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design
6. Supporting risk mitigation

7. Monitoring / verifying actual risk
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
23
Scope
CTA
TMA
CTR
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
24
Scope
ANSP service boundary
ANSP managerial domain
Hazards to be mitigated
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
25
Safety criteria
TLS (ICAO, ESARR4, ANSP)

1x10-7 obstacle collisions in

approach
TLS
1.55x10-8 accidents per flight

hour ATM direct contribution
ALS
4.10-7 accidents per flight hour in

overall ATM product
Qualitative - quantitative
no unacceptable hazards introduced
100
Absolute relative
positive safety effect
90
80
70
60
Scenario 1
Scenario 2
Scenario 3
50
40
risk should not increase
30
20
10
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
0
Low
High
26

assessment
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
27
Description of operation
CURRENT
OPERATIONS DESCRIPTION
NEW
FUNCTIONALITY
ENVIRONMENTAL
DESCRIPTION
AIRCRAFT
PERFORMANCES
HUMAN
ADJACENT CENTRES
CAPABILITIES
CURRENT CNS/ATM
CAPABILITY
AIRPORT
INFRASTRUCTURE
AVAILABILITY
FUNCTIONALITY
PERFORMANCES
PROCEDURE
EQUIPMENT
LOCAL WEATHER
PHENOMENA
TOPOGRAPHY
OBSTACLES
TRAFFIC
CHARACTERISTICS
ENVIRONMENTAL
CONSTRAINTS
NOISE SENSITIVITY
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
28

assessment
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
29
Hazard identification

Functionally
Brainstorming
ATCOs
And
Pilots
WHAT IF?
SAFETY
EXPERT
MODERATOR
SYSTEM
EXPERTS
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
30
Hazard identification
Technical hazards

Radio failure
Nuisance alert (erroneous alarms)
Operational hazards

Pilot misunderstands ATCO and takes off erroneously

System generates alert, but ATCO does not react
appropriately
Pilot on the wrong frequency
Pilot is mistaken/confused/lost and accidentally enters
runway
Pilot takes off without clearance
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
31

assessment
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
32
Conflict scenarios
cause 2
cause 1
hazard
Runway
unexpectedly
occupied
hazard
hazard
Aircraft on
final
approach
hazard
Detection & resolution

by ATC and flight crew
No safety
effect
Significant
Incident
Major
Incident
Serious
Incident
Accident
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
33
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
34

assessment
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
35
Severity assessment
ESARR 4 Severity Classes
Effect on
Examples of effect on operations include:
operations
1
Accident
Serious
Total loss of flight control
Mid-air collision
Controlled flight into terrain
Aircraft collide on the ground
Large reduction in separation without crew or ATC fully
Incident
controlling the situation
Aircraft deviating from intended clearance so that abrupt

manoeuvre is required to avoid collision with another aircraft or
terrain
Major
Incident
Large reduction in separation with crew or ATC controlling the

situation and able to recover
Minor reduction in separation without crew or ATC fully

controlling the situation
Significant
Incident
Increasing workload of the ATCO or flight crew, or slightly

degrading the functional capability
Minor reduction in separation with crew or ATC controlling the

situation
No safety
effect
No hazardous condition, i.e. no immediate direct or indirect impact

on the operations
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
36
Severity assessment
EFFECT CLASSIFICATION
Medium
Positive:
Large
Positve:
Effect can decrease the
probability of an accident

probability of a serious
incident
Small
Positive:
probability of an incident
Effect on Safety
and/or
can prevent an event,
which may lead to an
accident with other events.
Negative:
Negative:
Negative:
The effect can directly

cause a serious incident to
become an accident that
cannot be prevented
The effect can cause an

incident to become a
serious incident
The effect can cause an

incident
and/or
and/or
and/or
can increase the probability

of a serious incident
can increase the probability

of an incident
can introduce an event

which may lead to an
accident together with
other events
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
37
Severity assessment
Aircraft or vehicle
deviates from
intended route
and/or altitude
no traffic in neighbourhood
Minor incident
traffic in
neighbourhood
Conflict occurs
due to deviation
Hazard
Hazard
Hazard
Conflict is timely
detected and smoothly
solved
Hazard Hazard
Major incident
(condition: conflict occurs

due to error)
Hazard Hazard
Conflict is detected
no collision
and solved at last moment,
or not detected/solved
(condition: conflict occurs
due to error)
collision
Serious incident
Accident
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
38
Frequency assessment

historical data
data bases
interviews
expert judgment
simulation
Accidents
Incidents
Normal Operations
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
39
Frequency Assessment
1 of 100 departures
1 to 10 per year
5 to 10 per year
aircraft needs
more time for
backtracking
aircraft enters
rwy without
clearance
vehicle enters
rwy without
clearance
Runway
unexpectedly
occupied
20,000 approaches per year
2 minutes occupancy time
Aircraft on
final
approach
No safety
effect

2% of the time
visibility
radio failures
workload
1x10-4
Significant
Incident
Major
Incident
Serious
Incident
Accident
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
40
Risk tolerability
Frequency of
occurrence of hazard
effect (ST)
SEVERITY CLASS
Per flight-hour
-1
ST > 10
-3
-1
-5
-3
-7
-5
-9
-7
10 < ST < 10
10 < ST < 10
10 < ST < 10
10 < ST < 10
-9
ST < 10
10
10
Risk (in accidents/flighthour)
10
10
10
10
10
10
10
10
10
TLS
ALS
Risk vs. route spacing
-1
-2
*
o
+
x
-3
-4
TLS
(Eurocontrol)
* without
ATC
without ATC
o cur rent ATC
current ATC
+
Initial
Free
Flight equipped
initial Free Flight
- TLSded
(Eurocontrol)
exten
Free Flight
x Tar get Free Flight equipped
-5
-6
-7
-8
-9
target
-10
-11
10
15
20
Spacing S (in km)
25
30
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
41
Risk matrix
Risk = Frequency x Severity

Frequency of
occurrence of hazard
effect (ST)
SEVERITY CLASS
ST > 10-1
10-3 < ST < 10-1
10-5 < ST < 10-3
10-7 < ST < 10-5
10-9 < ST < 10-7
ST < 10-9
Per flight-hour
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
42

assessment
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
43
Risk Mitigation
aircraft needs
more time for
backtracking
aircraft enters
rwy without
clearance
vehicle enters
rwy without
clearance
Bottlenecks?
Runway
unexpectedly
occupied
Aircraft on
final
approach

visibility
radio failures
workload
No safety
effect
Significant
Incident
Major
Incident
Serious
Incident
Accident
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
44
Risk mitigation
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
45
Risk mitigation
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
46

assessment
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
47
Monitoring and verification

90
80
70
60
50
RWY 06
40
RWY 14
30
20
10
0
1st Qtr
2nd Qtr
3rd Qtr
4th Qtr
SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
48
In summary...
ITERATION
5. Evaluating risk
8. FEEDBACK
to operations,
assessment
and design

SA
FACULTY OF T
TRANSPORT AND TRAF
FFIC ENGINEERING
49

Safety Assessment Methods

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Safety Assessment Methods

Transféré par

Droits d'auteur :

Formats disponibles

DIVISION OF AIRPORTS AND AIR TRAFFIC SAFETY

FACULTY OF TRANSPORT AND TRAFFIC ENGINEERING

Among the class of complex and safety critical industries, air

In addition, from an organisational perspective, air traffic involves

The implication is that safety of air traffic is the result of interactions

Since safety depends crucially on the interactions between the

It is imperative to understand the safety impact of these

Methods/Models of the Risk and

Causal methods/models for risk and safety assessment of aircraft

Collision risk methods/models are dealing with assessment of the

Methods/Models of the Risk and

Human factor error methods/models. This category deals with

Third-party risk methods/models. This category of

Causal methods/models for the

Causal methods/models of assessment of risk and safety of aircraft

These methods/models can be qualitative or quantitative.

The former provide a diagrammatic or hierarchical description of the

The later estimate the probability of occurrence of each cause and

Causal methods/models for the

In addition, they can estimate the relative benefits of different

Some of the methods/models are as follows:

Fault Tree Analysis (FTA);

Collision risk methods/models (1)

One of the principal matters of concern in the daily operation of civil

Although aircraft collisions have actually been very rare events

In general, separating aircraft using space and time separation

Collision risk methods/models (2)

However, due to reduction of this separation in order to increase

The Reich-Marks model;

Human factor error

Investigation of causes of particular air traffic accidents has

Human error is considered as an incorrect execution of a particular

Therefore, monitoring and modeling of human errors in the aircraft

Human factor error

Consequently, many methods for detection and prevention of

HAZOP (Hazard and Operability) method;

Third-party risk methods/models

Third-party risk implies risk if an individual on the ground to be killed

Modeling of third-party risk has shown promise in resolving these

Design approaches (1)

Traditional ATM design approaches tend to be bottom-up, that is

The advantage of the traditional approach is that advanced design

The disadvantage of this traditional approach is that it fails to fully

Design approaches (2)

A goal oriented approach would be to design ATM such that safety

An early guidance of ATM design development on safety grounds

Although understanding this idea is principally not very difficult, it

This feedback should provide information on which safety-capacity

Design approaches (3)

A safety risk assessment methodology which provides safety risk

Generic Safety Assessment Process (1)

Safety assessment is the process through which it is assessed

This means that all possible impacts of a new operation or system

A safety assessment considers the proposed operation or system

Generic Safety Assessment Process (2)

Safety Assessment of an air traffic operation can be seen as a

The first refers to Iteration, meaning that safety assessment is usually

The second feedback loop is safety communication and feedback

Generic Safety Assessment Process (3)

Generic steps of a safety

6. Supporting risk mitigation

ANSP managerial domain

TLS (ICAO, ESARR4, ANSP)