Characteristic-based risk assessment and its

relation with event-based risk assessment

Thijs Mathot
Nuzo
Strukton
Mail@ThijsMathot.nl

Rob Hamann
Sec
2

rjhamann@ipact.nl

Eberhard Gill
Space Systems Engineering,
Faculty of Aerospace Engineering,
Delft University of Technology
E.K.A.Gill@TUDelft.nl

Copyright 2012 by Thijs Mathot. Published and used by INCOSE with permission.

Abstract. Common risk management practice is based on events. Event-based risk
management consists of identifying and mitigating the most significant events that could
possibly occur during a project and will have a negative impact on the key performance
parameters, development schedule and cost of the product if they occur. The shortcomings of
this approach are that this list of risk events is never complete, and is only partially available for
the early important design decisions. The proposed solution is to complement event-based risk
management with a so-called characteristic-based method. This second method estimates the
total risk of the project based on project and product characteristics. The advantages of the two
separate methods not only complement each other, but it will also be shown that they enhance
each other.

Introduction

Risk in a space project materializes through undesirable events with a negative impact on the
project when they occur. The possible occurrence of those events is either identified in advance
or remains unidentified until the occurrence. An example of such an event would be the
discovery of errors in the control software in the final testing phase. The possible impact of
these events is usually defined as an impact on the budget, schedule and/or performance of the
product/mission. An understanding of this risk in advance provides an opportunity to reduce
their significance and hence improve the project result. There are two ways to approach this
risk and accomplish the desired improved project outcome:

Event-based risk management: This approach focuses on the possible outcomes of the risk; it
is an attempt to identify all possible undesirable events through which the risk could
materialize. The aforementioned discovery of errors in the control software is an example of
such an event. To reduce project risk, a long list of these events is ordered according to
probability and impact. The events that are classified with the highest probability and impact
combination are taken into consideration for mitigation. Mitigation in this case consists of
trying to prevent these risk events from occurring, or reduce their impact on the project
outcome if they occur.




Characteristic-based risk management: This second approach focuses on the source of the
risk instead of the possible outcomes. It is possible to identify and remove these sources of risk,
or make an informed decision to accept the risk. Possible sources of risk that could lead to the
errors in the control software from the previous example, are technical difficulty or the lack of
experience of the project team.

The purpose of this paper is to show that both approaches do not only complement each other,
but enhance each other in a way that results in a better total risk estimate. The fact that the
combination is more than the sum of the individual parts could be a good reason to adopt both
methods in projects. The paper is divided in three parts. It starts with a discussion on the risk
events, followed by a discussion on the source of these risks, after which the combination of
both approaches is discussed.

Risk event approach: Taking care of smooth operations

The discovery of errors in the control software, or a support structure that collapses during
launch, but also a subcontractor going bankrupt, all three are risks that could be identified as
events that may occur during the remainder of the project. The common way to address these
events is to classify both the probability and impact on a subjective scale (for example: 1: low,
2: medium, 3: high, 4: very high) and ascribe to the events with the highest classification a
mitigation plan [ECSS, 2004]. Approaching project risk in this way is very common, and for
good reason: it is easy, straight forward, and if done properly, this approach will result in
smooth operations. However, it does not take into account project and product characteristics.
In the next section it will be argued that there are certain specific characteristics that are at the
cause of these events and thus require consideration as well in order to achieve an optimal
project result. In addition, it is impossible to get a complete risk assessment by just considering
the possible events because the list is infinite. And finally, especially early in the project when
the most important defining decisions are made, little event risk information is identified and
available to take into account in these decisions.

The relation between 1) the classification of probability and impact on a subjective scale (1:
low, 2: medium, 3: high, 4: very high) and 2) the real quantitative probability and impact (in
dollars budget overrun, days of schedule overrun, and percentage in performance degradation
for example) can best be approximated by an exponential relation as described by equation [1]
[Deutz, 2008] and an example can be found in [Hall, 2011]. By way of example, we might
consider the following exponential schedule impact classification: a low schedule overrun
could be a few hours, a medium overrun a few days, high could correspond to two weeks and
very high to 2 months. A low probability could mean an occurrence of 1 in 1000 projects,
medium 1 in 100 projects and high 1 in 10 [Ship Structure Committee, 1994]. The risk of an
event that is assessed can therefore be expressed as:


W H Z F W H Z F
e G E e G e E I P R
+
= = =

[1]

In equation [1], P and I indicate the quantitative real probability and impact of the risk
respectively. Z and W are the subjective classifications (i.e. 1 for low, 2 for medium, 3 for high
etc.) of the probability and impact, and finally E, F, G and H are unknown constants.

Based on equation [1] an interesting interim conclusion can be drawn. Since a high
probability/high impact event is a high risk event and low probability/low impact a low risk


event, and because the relation between the qualitative classification and quantitative real
probability and impact is exponential, the relation between the qualitative classification and
actual quantitative risk is also exponential.
It is common practice to multiply qualitative classifications of probability and impact in order
to determine a qualitative risk level. Because a linear qualitative scale represents an
exponentially increasing quantitative real risk level (
W H Z F
e G E
+
), equation [1] shows that
instead of multiplying qualitative classifications, summing them ( W H Z F + ) provides a better
qualitative representation of the actual quantitative risk.
Risk characteristics approach: Especially useful in the important
early decisions

The underlying idea of the theory described in this paper is that all possible risk events,
assessed as described in the previous section, could have been prevented if people involved in
the project would have made different decisions. The fact that if they occur they could have
been prevented means that someone involved in the project made a less than optimal decision.
It is not the intention to claim here that all events occurring that have a negative impact on the
project outcome could reasonably have been foreseen and prevented, nor that it is a good
approach to try to prevent them all. It simply means that all negative impacts on the project
outcome are the result of human actions that, from hindsight, where less than optimal.
Research in human behavior has identified the variables that determine human performance in
a general task [Swain and Guttman, 1983]. The first variable is the specific difficulty, i.e. the
ratio of task difficulty and executor capability, describing how difficult the task is for the
person that executes it. The second one is the pressure on the executor, a ratio of amount of
work and available time. A project wide analogy for this relation is described by the
characteristics quotient (CQ) in equation [2] given below, the number of less than optimal
decisions made in a project depends on the work difficulty (WD), project team capability (TC),
amount of work (AW) and available resources (AR). To which degree CQ depends on these
variables is unknown, for that reason b
1
to b
4
are undetermined constants. If a project is split in
different parts, being it product subsystems or project phases for example, CQ as described in
equation [2] provides a possibility to compare the number of undesirable events (i.e. risk events
as defined in the previous section) that can be expected to occur per fixed quantity of work.

4
3
2
1
b
b
b
b
AR
AW
TC
WD
CQ = . [2]

The number of decisions that result in a negative impact on the project is larger in a more
difficult project (higher WD), or bigger project (higher AW). A more capable team (higher TC)
reduces this number of decisions leading to materializing risk, as does more money, time,
people, or other resources (higher AR).
How can equation [2] be used to compare the difficulty of making optimal decisions for
different project parts? Or how can equation [2] be used to compare the risk involved with the
development of different design options? The first step is to identify more specific descriptive
project characteristics that determine the work difficulty, team capability, amount of work, and
available resources. The list in table 1 as identified by [Bijl and Hamann, 2004] for space
projects can serve as an example and starting point. For example, the requirement stability is an
aspect that partly determines the difficulty of the work (WD) the project team has to complete.
State of technology is an aspect that determines how much development work (AW) remains to
achieve a working product.


Table I: The variables determining the number of risk events and the corresponding
item and project characteristics. [Bijl and Hamann, 2004]
Work difficulty Team Capability Amount of work Available resources
Requirement
stability (RS)
Technical
difficulty (TD)
Manufacturing
difficulty (MD)
Assembly
difficulty (AD)
Material
resources (MR)


Sub-contractor
experience (SE)
Sub-contractor
relation (SR)
Customer
experience (CE)
Customer
relation (CR)
Personnel
experience (PE)


State of
Technology (SoT)

Personnel
resources (PR)
Test
Resources (TR)
Equipment
resources (ER)


Equation [3] is a possible combination of equation [2] and the variables in Table I (for different
examples see [Bijl and Hamann, 2004]). The combination results from the idea that
Requirement Stability (RS) to Material Resources (MR) each make an independent
contribution to the work difficulty (WD), Sub contractor Experience (SE) to Personnel
Experience (PE) make an independent contribution to the Team Capability (TC), State of
Technology (SoT) to the Amount of Work (AW) and finally Personnel Resources (PR) to
Equipment Resources (ER) contribute to Available Resources (AR). Weight factors as well as
values between 0 and 1 can be ascribed to the project characteristic variables RS to ER
according to predetermined ranking tables, for examples see [Mathot, 2008]. With all the
relevant project characteristics included in equation [3], c
1
to c
4
must represent a combination
of irrelevant characteristics and can thus be assumed to be negligible. Furthermore, to come to
equation [3], the impact of project difficulty, team capability, amount of work, and available
resources, or b
1
to b
4
, are similar and thus result in the single constant B in equation [4].


( )
( )
( )
( )
4
3
2
1
4
3
2
1
b
b
b
b
c Er Tr r P
c SoT
c PE CR CE SR SE
c Mr AD MD TD RS
CQ
+ + +
+
+ + + + +
+ + + + +
= [3]

Using equation [3], CQ in combination with a constant A can be used to estimate the total
number of events that could occur during a project. However, to provide an indication of the
total level of risk, also an at least qualitative estimate of the average impact of these events is
required (negligible, low, medium, or high impact). Use of the RISK$ assessment tool has
shown that experienced engineers are very capable of providing accurate distinctive qualitative
estimates of this average event impact [Hamann, 2007] for different project phases, product
parts or design options. Following the same rationale as in the previous section, the relation
between the qualitative average event impact assessment (Y) and the quantitative real impact is
exponential. The resulting risk based on the to-be-expected number of occurring risk events
and their average impact can thus be described by:


Y D B
e C CQ A R

= [4]

For comparing different design options, or assessing in which part of the project the largest
risks can be found, the constants A and C are of no importance because they are similar for


every option and project part. The constants B and D can, without sufficient statistical data to
prove otherwise, be selected in such a way that the complete range of possible values for CQ
and Y has a similar influence on R and are thus, for a qualitative comparison, also not required.
The added benefit of not only assessing the risk events but also assessing the project
characteristics that determine the project risk, is that it provides a structured approach to
dealing with risks from the beginning of the project. Moreover, it highlights where risk is
coming from: which part or phase, but also which characteristic of this part or phase. These
characteristics make the method especially suitable for taking risk into consideration when
comparing different design options in a technical trade-off. It offers the possibility to eliminate
risk early in the project when it is still possible to make significant changes, or make an
informed decision about accepting risks. These options are not provided by the event-based
method that only offers an approach to deal with already existing risk later in the project.
A combination: the theory of a combined risk strategy

The previous two sections indicated the complementing applications of both methods. The
purpose of this section is to describe the theory of a combination of both methods. Since the
risk described by equation [4] is by definition the total project risk, it is equal to the sum of all
individual event risks. Furthermore, also the probability of the individual events occurring and
the number of events that can be expected to occur are related, two equations describe the event
and total risk relation:

=

=
r
k
W H Z F
n
Y D B
n
k n k n n
e G e E s e C CQ A
1
* *
*
. .
*
[5]

|
|
.
|

\
|
=
[
=

r
k
Z F
n
B
n
k n
e E
s
CQ A
1
.
1 ln [6]

If a project or product is divided in 1 to n parts (e.g. phases or subsystems) equation [5]
describes on the left side of the equal sign the total number of events to be expected for project
part n multiplied by their average impact (equation[4]). The right side of equation [5] sums the
probability and impact for the k=1..r identified events for project part n. Equation [6] describes
the relation between the expected number of events and the individual event probabilities. As
discussed before, there are multiple types of impact and hence multiple types of risk: budget,
schedule, and mission performance risk could be assessed for example. The asterisk in
equation [5] can be replaced with either b, s or p to indicate these three different types of risk.
This means that the risk involved with project part n is described by four equations: three times
equation [5] and once equation [6].

In equation [5] and [6] s
n
is a multiplication factor to compensate for the problem introduced in
section one: it is impossible to assess all possible events. What does this variable s
n
describe in
practice? Let us discuss an example: Two project parts, part 1 and part 2, are compared with a
combination of both assessment approaches. According to the characteristic-based assessment,
part 1 constitutes high risk, part 2 constitutes low risk. However, for part 2 there are more risk
events and higher risk events identified. Comparing equations [5] and [6] for both parts will
show that the variable s
1
will be higher than s
2
. The higher s
1
provides an indication that the
event assessment of part 1 is less complete than for part 2 (or there are errors in the existing
assessment). Even without sufficient statistical data to determine values for the constants A to
H, such comparisons of the variable s
n
can be made in a qualitative manner by eliminating the
constants in equation [5] and [6] through devision .



Since for risk events it is so accurately described what goes wrong if they materialize, it is not
difficult to quantitatively estimate the probability of their occurrence nor their cost, schedule
and performance impact. In addition, a random error in these assessments cancels out over the
large number of events. The problem is that the project risk based on these event assessments is
never complete because the event assessment is never complete. A best fit solution to equation
[5] and [6] for multiple project parts based on quantitative probability and impact estimates for
the events can thus provide an additional evaluation of the total risk estimates that in addition
takes into consideration the risk events that have not been identified.

Figure 1 is a graphical presentation of an example constructed for the purpose of this paper in
which both assessment methods have been combined. Assume a new mars rover needs to be
developed and the propulsion system needs to be selected. The considered options are 1:
nuclear, 2: electrical and 3: liquid fuel. Based on the project and product characteristics, the
risk levels involved with the development and production of the propulsion subsystem have
been determined at the earliest stages of the project. These risk levels are depicted by the black
bars on the left side of the graph. Because the characteristic-based risk level is available early in
the project, it can be taken into consideration for the trade-off (not leading necessarily to the
selection of the lowest risk option). Also conscious modifications to the design, team or project
can be made based on a sensitivity analysis of the sources of the risk. In the example, this has
resulted in the selection of the electrical option 2 with some risk reducing modifications (the
fourth black bar in the graph).


Figure 1: This screenshot from the assessment tool RISK [Mathot, 2008] depicts the
level of risk that is determined by the combination of both assessment methods as
described in the example. The black vertical bars represent the risk as determined
using the characteristic based method where through a trade-off between three
options, a modified option 2 has been selected. The waterfall chart depicts the
different risk events whose influence is reduced over the course of the project. As
indicated by the hashed area, not all events are identified from the start of the project.

This total development and manufacturing risk of the electric propulsion system, as determined
based on the item risk, is equal to the total risk of all possible events that could negatively
impact the development and manufacturing. Even though these events are not yet identified, as
depicted by the black hash, this total risk level is known. The waterfall graph depicts how more
risk events get identified over the course of the project, shown by the receding hash, and how
the identified events are used to take risk reduction measures (aimed at reducing the probability
and impact of these events). Through these risk reduction measures, and by risk events
occurring or passing without occurring, risk is gradually reduced until the risk level reaches
zero at the end of the project.


Conclusion

The risk assessment approach commonly practiced identifies the events that could occur in a
project and if they occur would have a negative impact on the project outcome. An alternative
approach is to assess the project risk through the classification based on a number of relevant
project and product characteristics. While the event-based method describes the project risk
based on the possible risk outcome, the characteristic-based method assesses the risk based on
the causes. The most important advantage of event-based risk assessment is that it helps
smoothing operations as it allows to focus on preventing the events from occurring or reducing
their impact. The most important advantage of the characteristic-based method is that it
supports high impact decisions early in the project as the required information is available
earlier in the project and there is a direct link between the decision to be made and its influence
on the project risk. Next to supplementing each other, combining both methods allows more
reliable estimates of the total project risk through a combination of both assessments. The main
benefit of this more accurate total risk estimate is that it provides a better estimate of the total
programs likely amount of risk reserve funds needed, the amount of likely schedule delays, and
the amount of likely performance degradation. Statistically significant quantitative results will
require performance data on previous projects. On a qualitative basis the combined approach is
ready to be implemented and tools to support it are available.

References

Bijl, M.G.H., Hamann, R.J. 2004. User Requirements for a Technical Risk Assessment
Technique Proceedings of the 14th Annual International Symposium and the 4th
European Systems Engineering Conference of INCOSE, (Toulouse, France)

Deutz, A. 2008. Personal interview with Dutch Space Systems engineer (Leiden,
Netherlands)

European Cooperation for Space Standardization (ECSS) 2004Space project management,
risk management ESA-ESTEC (Noordwijk, Netherlands)

Hall D.C. 2011. Making Risk Assessments more Comparable and Repeatable Systems
Engineering, Vol. 14, No. 2 (Hoboken, US)

Hamann, R.J. 2007. Personal interview with Systems engineering specialist at the faculty of
aerospace engineering Delft University of Technology (Delft, Netherlands)

Mathot, T. 2008. Risk management in the space industry TU Delft (Delft, Netherlands)

Ship Structure Committee. 1994. The role of human error in design, construction, and
reliability of marine structures National Technical Information Service (Springfield,
US)

Swain, A.D.; Guttman H.E. 1983 Handbook of human reliability analysis with emphasis on
nuclear power plant applications US Nuclear Regulatory Commission (Washington,
US)

Whalen J., Wray R.B., Mc Kinney D. 2000. Systems Engineering Handbook, A "how to"
guide for all engineers INCOSE (Seattle, US)


Biography

Thijs Mathot received a Masters degree in Aerospace Engineering at Delft
University of Technology. Risk management in the space industry was the
subject of his Thesis. He also received a Masters degree in International
Business at ESCP-EAP in London. Since 2010 he works for Strukton on
multidisciplinairy construction projects in the Netherlands and is a partner at
Nuzo real estate projects. (mail@thijsmathot.com)

Eberhard Gill, born 1961 in Germany, received a diploma in physics and
holds a PhD in theoretical astrophysics of the Eberhard-Karls-University
Tuebingen, Germany. He holds a Master of Space Systems Engineering of the
Technical University Delft.
He worked as a researcher at the German Aerospace Center (DLR) from 1989
to 2006 in the field of precise satellite orbit determination, autonomous
navigation and spacecraft formation flying. He has been co-investigator on
several international missions, including Mars94-96, Mars-Express, Rosetta, Equator-S and
Champ. Dr. Gill acted as Principal Investigator on the PRISMA formation flying satellite
mission and has developed a GPS-based onboard navigation system which is operating on the
BIRD micro-satellite since 2001. Since 2007, he holds the Chair of Space Systems Engineering
of the Delft University of Technology. In this framework, the chair has developed the
nano-satellite Delfi-C3 which was launched in 2008 and is successfully operating since then.
The research interests of Prof. Gill are spacecraft autonomy, formation flying, GNSS
technology, and combined communication and navigation applications.
Dr. Gill has authored or co-authored more than 150 journal articles and conference papers. He
has also co-authored three text books: Relativity and the Earth's Rotation (Springer 1990),
Satellite Orbits (Springer 2000) and Applied Space Systems Engineering (2010). Dr. Gill holds
a patent in the field of Galileo and GNSS Applications. He has been awarded Senior Scientist
of DLR in 2006. He acts as research advisor for several universities and research institutions. In
2008, he was appointed program director of the SpaceTech post-graduate program of the Delft
University of Technology. He is corresponding member of the International Academy of
Astronautics. (E.K.A.Gill@tudelft.nl)

Rob Hamann received an Aerospace Engineering education at Delft University
of Technology in the Netherlands and Princeton University Graduate School,
USA. From 1974 to 2000 he worked at Dutch Space (formerly Fokker Space),
Leiden, the Netherlands as an Engineer and Systems Engineer for many space
projects. From 1990 he has been in charge of introducing, developing and
maintaining the Systems Engineering methodology at Fokker Space. Since that
time he has been a visiting lecturer on the subject at the Delft University of Technology,
Twente University in the Netherlands, and the Ecole des Mines de Nantes in France. From
2000 until 2009 he has been employed at the Delft University of Technology as Coordinator
Space Systems Engineering and Senior Lecturer. He has been the Project Manager of the first
Dutch University satellite Delfi-C
3
, launched in 2008 and still operational. He has authored and
co-authored more than 40 journal and conference papers.
Since July 2009 Rob is an independent SE consultant with SEc
2
and affiliated to Delft
University of Technology. He is currently a member of the European Sentinel 5p satellite Joint
Project Team at ESTEC. (rjhamann@ipact.nl)