METHODS AND METRICS FOR DESIGN OF VALUE ROBUST SYSTEMS C. Malan1* and L.

Pretorius2 Department of Engineering and Technology Management University of Pretoria, South Africa cmalan@csir.co.za Department of Engineering and Technology Management University of Pretoria, South Africa leon.pretorius@up.ac.za ABSTRACT Systems acquisition is faced with three challenges in the short term: performance, schedule and cost. However, during the life cycle of the system a new challenge in the form of change becomes prominent. This may be change in needs of the stakeholders or change in the context in which the system is applied. The perceived value of the system is often reduced by these changes. The system that was optimized for the short term is very soon sub-optimal if it is not robust to change. The challenge therefore is not to optimize the system design for the short term, but to optimize the system to provide value over its life cycle. Value robust systems have the ability to continue to provide value to stakeholders when subjected to change in context or demand on the system. The system design methodology required for this design problem is the subject of this research. This paper provides an overview of current ideas and of intended research into design principles and supporting metrics for value robust system design.

1. INTRODUCTION This paper provides an overview of current ideas and of intended research into design principles and supporting metrics for value robust system design. The design and engineering of systems is by nature a multi-objective optimisation problem in that the four primary objectives of cost, schedule, performance and risk have to be optimised in terms of the perceived value of stakeholders of the system. These objectives are often in conflict and improvement in one objective result in a compromise in one or more of the others. During the system development phase, trading cost and schedule is the responsibility of the project manager, while trading performance, cost and risk is in the domain of system engineering as shown in Figure 1.
Maximize Performance

Minimize Schedule

SYSTEM DEVELOPMENT

Minimize Risk

Minimize Cost

Figure 1: Primary objectives for system design optimisation During the life cycle of a system, the context in which the system is applied will change and the needs of the users of the system will change. These changes result in a reduction in the perceived utility provided to stakeholders by the system, unless the system is designed to be robust in the face of these changes. The need be robust to change over the long term places a new perspective on the short term design objectives of cost, schedule, performance and risk considered during the development phase. New design criteria, such as flexibility, adaptability, scalability and versatility are recognised as desirable properties of a system and are essential to the long term success of systems. These system attributes are collectively referred to as ilities and are difficult to analyse in isolation as they are not defined as independent system attributes, but as an expression of a combination of system attributes under varying contexts over time [1]. A framework is

The author was enrolled for an PhD Eng degree in the Department of Engineering and Technology Management, University of Pretoria. 2 The author was research leader in the Department of Engineering and Technology Management, University Pretoria. *Corresponding author South African Journal of Industrial Engineering Month Year Vol __(_): p1-3

required within which these attributes may be defined and analysed to gain insight into the design principles that make systems adaptable, flexible, upgradable and scalable. Two broad strategies have been employed to directly incorporate stakeholder value into the design optimisation process and thereby incorporate the design criteria of ilities in the system design: MultiAttribute Utility Theory which is based on decision theory and Value-driven Design which is based on microeconomic theory. The distinction between the two approaches lies in how stakeholder value is modelled (de Weck) [2]. Multi-Attribute Utility Theory uses system value attributes to express the utility that may be derived from a given system design. This expression of value is the objective function which is optimised through the system design process. Value driven design (VDD) on the other hand uses microeconomic analysis to build a value model which is most often expressed as a monetary value. However, the value of a noncommercial system is not easily expressed as a monetary value and leads to difficulty in the application of VDD to systems for government, military or research applications (Collopy) [3]. In both design approaches, system value is optimised and not primarily system performance. These strategies lead to two value based approaches to system design optimisation which incorporate the value based design criteria, such as flexibility, adaptability, scalability and versatility, (the ilities). Central to both of these approaches are methods of system analysis which depend on models and metrics to characterise useful attributes of a design. This paper explores a range of metrics which will support these methods and in particular metrics which may assist in design of systems which are robust to change over the system life cycle. The Multi-Attribute Utility Theory approach is identified as a useful design approach for systems designed for use in non-commercial environments. A literature review of design metrics supports research into a design methodology based on Multi-Attribute Utility Theory for systems subjected to change. A decision may be made to limit the investigation to an evolutionary acquisition model. This paper consists of four sections. After the introduction, the second section introduces Multi-Attribute Utility Theory as an approach for system value robust system design. The third section provides an overview of the research focus and design. The fourth section provides an overview of current research into deign principles and metrics applicable to ilities. 2. Multi-Attribute Utility Theory System Design Approach for ilities The optimisation of the primary objectives in Figure 1 will lead to an optimum point design for the system. Stakeholder value is not one of the primary objectives in the design optimisation process shown in Figure 1. A cost benefit ratio may be applied to a number of point designs to select a design which provides the best ratio of benefits and cost. The decision to select between point designs is then determined by cost, and the optimisation therefore treats cost as an independent variable (CAIV) in the multi-objective optimisation problem (Dahlgren) [4], as shown in Figure 2. Inherent in the design optimisation process described above is a system performance model, a system cost model and a value model which provides an assessment of the stakeholder value derived from the system.
Utility Pareto Frontier (for optimum solutions) Trade with CAIV

Best value solution may be here Solution Trade Space

Cost

Figure 2: Trade with Cost as Independent Variable (CAIV) Vifredo Pareto (1848-1823) created an economic theory referred to as the Pareto Optimum (Pareto) [5] for the optimum allocation of resources in a society. This theory was successfully applied to engineering problems and in particular to multi-objective optimisation problems in the engineering and design of complex systems (de Weck) [2]. Pareto analysis and the use of a Pareto frontier refer to a set of designs which are optimal (Agrawal , Bloebaum and Lewis 2005) [14]. Pareto analysis is useful for short term objectives in design where limited change is expected, but later in the system life cycle more flexibility is required to deal with new demands on the system. This was illustrated in the design of a satellite constellation which required staged deployment (de Weck, Hewitt and Chaize) [6]. The design of the satellite constellation required the incorporation of flexibility to cater for potential future expansion, at additional cost and mass. The preferred design was therefore not on the Pareto front. Pareto analysis remains useful to indicate how technology infusion and obsolescence impacts 2

on the design space and may indicate if a new technology will be successful in system context (de Weck, Chang) [7]. Value systems, of which Multi-Attribute Trade Space Exploration (MATE) is one example, are widely used in aerospace system acquisition to select a system design which will provide value to the stakeholders [27], [28]. The objective of trade space exploration is to avoid point designs and to explore the cost and utility of a larger trade space of possible designs. This process is shown in Figure 3 and follows the following steps: (1) Elicit Value from Stakeholders with attributes and a utility metric per attribute, (2) Generate design concepts based on system purpose or mission by specifying design variables, (3) Develop models to assess designs in terms of cost and utility, (4) Create a trade space by using multiple design vectors, (5) Explore the trade space select most valuable system design.
STATIC MATE PROCESS

Mission Concept

Attributes

Define Design Vector

Develop System Model

Calculate Utility

Estimate Cost

Utility Architecture Trade Space

Cost

Figure 3: Multi-Attribute Trade Space Exploration (MATE)

Context 1 As context changes from C1 to C2, System 1 fails to meet expectation E1, while System 2 continues to meet expectation E1. C2 B Z Expectation 1 X S1 E1 S2 System Epoch 1 Expectation S2 S1

Context 2

Context 2

Context 3 System upgrade

Context 4

Context

Expectation 4 Expectation 3 Expectation 2

C1

Expectation 1

Epoch 2

Epoch 3

Epoch 4

Epoch 5

Figure 4: Change in system context and system response (adapted from [8]) Ross and Rhodes [1] differentiate between a static and a dynamic Multi-Attribute Trade Space Exploration. The dynamic Multi-Attribute Trade Space Exploration process allows change in the demand on the system (change in context or change in mission requirements) to impact on the utility as shown in Figure 4. If a new requirement is placed on a system, the utility of the system may be much lower. Dynamic trade space exploration allows a system design to be analysed over a projected future time frame under conditions of change. The changes are introduced as discrete steps using an epoch-era construct shown in Figure 4. The static MATE is then applied in each epoch as the demand on the system remains static during the epoch. The era is a series of epochs providing a timeline of events over the system life cycle as shown in Figure 5.

STATIC MATE PROCESS

DYNAMIC MATE PROCESS

Mission Concept

Attributes

Utility Perform STATIC MATE Attributes Define Design Vector Designs Cost

Develop System Model Define Epochs Calculate Utility Context Estimate Cost Expectation

Utility
Epoch 1

Utility
Epoch 2

Time

Time

Utility Utility Architecture Trade Space Construct Eras Epoch Series Dynamic Strategies
Epoch 1 Epoch 2 Epoch 3 Epoch 4

Time

Cost

Figure 5: Static and Dynamic MATE Process (adapted from [8]) Ross [9] notes that value robustness may be achieved through a system design that continues to provide value even after a change in requirements without engineering change to the system (probably through an over design) or through adaptation of the system which requires an engineering change. Research by Ross and Rhodes [1] suggests two approaches to achieve value robustness and proposes applicable selection criteria in each case: (1) Select a design that provides high utility under dynamic analysis conditions (quantified by Pareto analysis) and (2) select designs that are changeable and can deliver high utility when needed (quantified by stakeholder tolerance to cost impact of changes required to adapt the system when required). The system design decisions made during the concept development phase of system acquisition will have considerable influence on the system, eventually enabling or limiting success of the system over time. Many of these design decisions are made while insight into the system design is still increasing and each decision made commits cost and constrains the design [10]. Designing systems for value robustness suggests that more emphasis should be placed on conceptual design and detail design to achieve value robustness over the system life cycle as shown in Figure 6.

Management Leverage

Cost Committed Knowledge Cost Incurred

Management Leverage

Cost Committed Knowledge Cost Incurred

Concept development

Detail Design

Production

Use

Concept development

Detail Design

Production

Use

Figure 6: Decision Impact through design knowledge increase (from [10]) The current research into Multi-Attribute Trade Space Exploration suggests that architecting systems for value robustness will require design principles and measurable metrics to guide design selection. 3. Research Focus and Research Design Beesmyer Fulcoly Ross and Rhodes [11] define evolvability as a design characteristic that facilitates more manageable transitions between system generations via the adaptation of an existing design. Ross and Rhodes [1] define a number of related ilities as follows: Versatility is the ability of a system to satisfy diverse stakeholder expectations without the need to change form and therefore expresses system inherent value to meet changing needs over time. Changeability is the ability of a system to alter form or function at acceptable cost to meet changing needs over time. Flexibility is defined as the ability of the system to be changed by an external change agent and Adaptability is defined as the ability of the system to be changed by an external change agent. These system attributes describe a system under change and suggests that design for value robustness should state a design objective for a system subjected to change over its life cycle. The 4

relationship between the design objectives for short term performance criteria and long terms value objectives need to be researched to gain an understanding of the value of complex systems over its lifecycle [2]. Much of the current research focus has been design and analysis approaches to define the role of one of these characteristics at a time. Suh and de Weck [13] presents a methodology to design for flexibility. Ross [9] presents a methodology to incorporate changeability into system design. Beesmyer et al [11] presents a metrics for design for evolvability in system design. Richards, Ross, Hastings and Rhodes [15] present design principles for design for survivability. It is conceivable that many of these properties may be desirable at the same time and metrics and methodologies devised for the individual ilities will be analysed concurrently and potentially use related metrics and design principles. A useful research framework is shown in Figure 7 highlighting the role of design principles and of metrics derived from [11]. Starting with the stakeholder mission objective (stated as a general value objective), the desired system characteristics including ilities is derived. Specific design principles should then be applied to realise the desired benefits through a design decision (related to architecture or design choice) which defines the design vector. The design vector defines the system point design in trade space and should be measured in terms of the design principle applied to the design. The metric informs on the achievement of the design principle.
Other Value Objective (Stated in terms of System Life Cycle Value not focused on short term objectives of performance vs Cost) Other Point Design System Design Vector

Value Metric (e.g. for Changeability)

Design Principles Attribute (e.g. Changeability) Principle (e.g. Modular Design)

Point Design

Design Metric (e.g. for Modularity)

Research Guidance ility Attribute leads to informs informs

Design Principles

leads to

Methodology

leads to

Design Metric

Figure 7: Research Guidance A mapping between system properties which result in value robust system designs (with a focus on appropriate selection of ilities), design principles and system metrics will make a valuable contribution to a combined design methodology for value robust system designs. This research will be guided by the framework in Figure 7. A brief overview of exploratory research into metrics follows, with focus on value robust system design and forms part of ongoing research within the framework shown in Figure 8, and is derived from [16], [17] and [18]. Step 1 performs exploratory research (descriptive how things are done) of current literature on value robust systems and will inform step 2 which will propose a new theory (normative - how things should be) for improvement of methods for design of value robust systems. Step 3 performs internal validity tests of processes using case study data from step 1 and Step 4 performs external validity tests using a representative case study. This step is prescriptive (how difference between descriptive and normative should be addressed) in nature to demonstrate the value of the research contribution.

Step 1: Research into Current Theories and Synthesis (Descriptive)

Current Theory

Step 2: New Theory Development (Normative)

Deductive Process

Design Process
Data Framework & Concepts Design Principles Trade space Methods & Metrics

Internal Validation

Step 3: New Theory Evaluation

Methods

External Validation

Step 4: Case Study Application (Prescriptive)

Methods Data

Inductive Process

Evaluation of Design Principles & Methodology

Case Study

Figure 8: Research Design Initial results of step 1 are provided below. 4. Literature review on metrics 4.1 System Attributes Value robust systems have the ability to continue to provide value to stakeholders when subjected to change in context or demand on the system. Change in context and demand considered here are changes over the system life cycle and not short term changes in environment associated with survivability, but on longer term value attributes such as flexibility, adaptability, scalability and versatility. Change in system context or stakeholder demand on the system may be small so that the system can cope with change through an inherent robustness to change, a system property called versatility. Alternatively the system may cater for larger changes by reconfiguration of the system to continue to meet the stakeholder value expectation, requiring a system property called configurability. If the change in system context or stakeholder demand is large, reconfiguration will fail to maintain system value and the system will have to be changed to meet the stakeholder value expectation, requiring a system property called changeability. System versatility and configurability enables the system to meet changed expectations without incurring cost, while system change incurs cost. The increasing extent of change imposed on the system will therefore require first versatility, then configurability and finally changeability to enable a system to cope with change. The desired properties are shown in Figure 9 and illustrates that a system design which is robust to change should ultimately limit cost incurred to meet changing stakeholder expectations. The scope of versatility, configurability and changeability may be based on expected change scenarios but will also encounter unplanned scenarios. The design principles and applicable metrics to design a system with these properties are discussed below.

Cost incurred to meet Stakeholder Expectation

Desired Desired Change in System Context or Stakeholder Demand

Versatility

Configurability

Changability

Planned

Unplanned

Planned

Unplanned

Planned

Unplanned

Figure 9: Scope of System Properties under Change conditions 6

4.2 Design Principles Suh, Kim, de Weck and Chang [13] investigated design principles for incorporating flexibility into design and then performed economic optimization of design options with various degrees of flexibility built in. It was found that design options with more flexibility performed poorer than design options with less flexibility built in. This stresses the fact that the ilities are incorporated into a design at a cost. The appropriate economic analysis approach is microeconomic options theory [4]. An option is a financial market contract that specifies the price at which the holder of the option can buy or sell an asset within a specific time frame. An option provides the right but not an obligation and is acquired at a price. The option may result in benefits depending on movements in the price of the asset within the specified timeframe, but not known when the option is acquired. It is normally used as a hedge strategy against risk (uncertainty). The analogy in system design is that ilities is a hedge against uncertain future change and should be incorporated at a reasonable cost. Siddiqi and De Weck [20] studied configurable systems and identifies design principles. These design principles apply the three ilities identified in the previous section as useful over the system life cycle. The design principles that they identify are: Configurability. For every configuration of a reconfigurable system, there exists a corresponding dedicated system that is at least equal in performance. A good reconfigurable design is one in which the performance of each configuration approaches that of the corresponding dedicated system. Self-Similarity. Systems with self-similar modules, have highest degree of configurability. Common modules should be maximized across configurations. Information Reconfiguration. Maximize the informational nature of the element under frequent reconfiguration. Reconfiguration costs of informational elements and interfaces is usually low.

Fricke and Schulz (2005 [21]) investigate changeability of systems and identifies the following design principles: Integratability. This design principle implies a modular design architecture and common standardized interfaces. Scalability. This design principle implies the ability to accommodate growth in capacity or number of system elements with low impact. Decentralization. This design principle is aimed at reducing change propagation through the system. Redundancy. This design principle enables the system to continue to perform when selected system components fail or degrade. This design principle effectively creates space capacity which supports versatility.

Consolidation of the design principles are needed as there is overlap between the principles described by various authors. This consolidation will also be supported by consolidation of metrics to be applied in each case. 4.3 Metrics Lave and March [19] proposed evaluation criteria for social science models and these criteria have been applied to system value models. These criteria may also be applied to metrics. The evaluation criteria are (1) truth (is the metric accurate), (2) beauty (is the metric elegant and easy to use) and (3) justice (will the metric use lead to sound design decisions). Though this evaluation is not applied in this study, the metrics will be evaluated at a future date. Responsiveness is a concept investigated by Dubos and Saleh [22] to investigate operational responsive spacecraft technology and impacts on changeability and acquisition models. From this investigation a metric related to system delivery schedule is derived to indicate that there is a time value to utility. If delivery of the system is delayed due to schedule slip or not available due to system engineering change, the stakeholders do not receive utility from the system. The metric calculates cumulative utility received over a time period and illustrates that a system design that is delivered in the short term and provides low utility may outperform a system design that is delivered much later and provides high utility. This metric may inform on design choice and on acquisition model. Technology Readiness Level (TRL), Integration Readiness Level (IRL), System Readiness Level (SRL) and Manufacturing Readiness Level (MRL) are investigated by Azizian, Sarkani, and Mazzichi [23]. They find that TRL is widely used (patricularly in the defence and space industry) but has many flaws due to its use to assess the technology state of a system or component and associated risk in acquisition without considering integration complexity or obsolecence. IRL provides insight into integration readiness by assessing interfaces 7

between system components. TRL and IRL combined in a matrix calculation leads to the SRL metric. These metrics have been incorporated into parametric cost models [24],and schedule models [22]. A design structure matrix (DSM) or dependency structure matrix is used to access interdependencies of system components, technology risk associated with internal interfaces and architecture of a system. Cai, Huynh and Xie [25] presents a metric for modularity of a design using DSM. Holtta-Otto [26] developed a metric to assess the degree of complexity of system interfaces. The metric is used in in combination with a modularization method to determine module boundaries and their complexity. The objective is to use this metric to guide system design towards solutions which limit change propagation and so minimize the redesign effort. The metric relies on interviews with multiple experts, which will ultimately limit its use. Further work is required to forge attributes, design principles, methods and metrics into a useful framework. 5. CONCLUSION Value robust systems have the ability to continue to provide value to stakeholders when subjected to change in context or demand on the system. Change in context and demand over the system life cycle indicate a combination of versatility, configurability changeability as desirable stakeholder values. Multi-Attribute Utility Theory and Value-driven Design are presented as two strategies to directly incorporate stakeholder value into the design optimisation process. The increasing extent of change imposed on the system will require system properties of versatility, configurability and changeability to enable a system to cope with increasing degrees of change. A common set of design principles and metrics are desirable to support the conceptual design of systems with these properties. Current research and practice in system conceptual design for value robust systems is used in a descriptive research process to identify useful design principles and metrics. The next step will be to map design principles to system attributes applicable to value robust systems (the selected ilities) and to use this mapping to identify a set of design methodologies which should be applied to achieve value robustness in systems over its life cycle. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] Ross, A.M. and Rhodes, D.H. 2008. Architecting Systems for Value Robustness: research Motivations and Progress, SysCon 2008 IEEE International Systems Conference Montreal, Canada, April 7-10, 2008. De Weck, O.L. 2004. Multiobjective Optimization: History and Promise, CJK-OSM3, 2004, Kanazama. Collopy, P.D. 2009. Aerospace System Value Models: A Survey and Observations, AIAA SPACE 2009 Conference & Exposition 14-17 September 2009, Pasadena, California. Dahlgren, J.W. 2006. Real Options and Value Driven Design in Spiral Development, 2006 CCRTS, The State of the Art and State of the Practice. Pareto, V. 1906. Manuale di Economia Politica, Societa Editrice Libraria, Milano, Italy, Translated into English by A.S. Schwier as Manual of Political Economy, Macmillan, New York, 1971. De Weck, O.L. de Neufville, R and Chaize, M. 2004. Staged Deployment of Communications Satelite Constallations in Low Earth Orbit, Journal of Aerospace Computing, Information, and Communication, Vol.1, March 2004. De Weck, O.L. and Chang, D.D. 2003. Quantitative Assessment of Technology Infusion in Communications Satellite Constallations, 21st International Communications Satellite Systems Conference and Exibit, American Institute of Aeronautics and Austronautics, AIAA-2003-2355. McManus, H.L., Richards, M.G., Ross, A.M. and Hastings, D.E. 2007. A Framework for Incorporating ilities in Tradespace Studies, American Institute of Aeronautics and Austronautics Space 2007. Ross, A.M. 2006. Managing Unarticulated Value: Changeability in Multi-Attribute Tradespace Exploration, Doctor of Philosophy Dissertation, Engineering Systems Division, MIT, June 2006. Rhodes, D.H. and Ross, A.M. 2008. SEAri Overview and Dynamic Multi-Attribute Tradespace Exploration, MIT-Portugal Transportation System Workshop, September 5, 2008. Beesmyer, J.C., Fulcoly, D.O., Ross, A.M. and Rhodes, D.H. 2011. Developing Methods to Design for Evolvability: Research Approach and Preliminary Design Principles, 9th Conference on Systems Engineering Research Los Angeles, CA, April 2011. Brown, O. 2007. Speech by Dr. Owen Brown on Fractionated Spacecraft, DARPATech Symposium. Anaheim, CA. Suh, E.S., Kim, Y., De Weck, O.L. and Chang, D. 2004. Design for Flexibility: Performance and Economic Optimization of Product Platfrorm Components. 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference 30 August-1 September 2004, Albany, New York. Agrawal, G, Bloebaum, C. L., Lewis, K. 2005. Intuitive Design Selection Using Visualized n-Dimensional Pareto Frontier, 46th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics & Materials Conference 18 - 21 April 2005, Austin, Texas.

[15] [16] [17] [18] [19] [20] [21] [22] [23]

[24] [25] [26] [27] [28]

Richards, M.G., Ross, A.M., Hastings, D.E., and Rhodes, D.H. 2008. Two Empirical Tests of Design Principles for Survivable System Architecture, INCOSE International Symposium 2008, Utrecht, the Netherlands, June 2008. Yin, R.K. 2009. Case Study Research Design and Methods Fourth Edition, SAGE Publications, Inc. Valerdi, R., Ross,A.M. and Rhodes, D.H. 2007. A Framework for Evolving System of Systems Engineering, CROSSTALK The Journal of Defense Software Engineering, October 2007. Rhodes, D.H. and Ross, A.M. 2008. SEAri - Architecting Systems for Value Robustness:Research Motivations and Progress, MIT, April 2008. Lave, C.A. and March, J.G. 1975. An introduction to social science in Social Sciences, Harper & Row, New York, 1975. Siddiqi, A and de Weck, O.L. 2008. Modeling Methods and Conceptual Design Principles for Reconfigurable Systems, Journal of Mechanical Design, Vol. 130, October 2008. Fricke, E. and Schulz, AP. 2005. Design for changeability (DfC): Principles to enable changes in systems throughout their entire lifecycle. Systems Engineering. 2005;8(4):342-359. Dubos, G.F. and Saleh, J.H. 2011. Spacecraft technology portfolio: Probabilistic modeling and implications for Acta Astronautica 68 (2011) 1126-1146. Azizian, N., Sarkani, S. and Mazzichi, T. 2009. A Comprehensive review of Maturity Assessment Approaches for Improved Decision Support to Achieve Efficient Defence Acquisition, Proceedings of the World Congress on Engineering and Computer Science 2009 Vol II WCECS 2009, October 20-22, 2009, San Francisio, USA. Valerdi, R. and Kohl, R. 2004. An Approach to Technology Risk Management, 1st Annual MIT Engineering Systems Division Symposium, Cambridge, MA. Cai, Y., Huynh, S., and Xie, T. 2007. A framework and tool supports for testing modularity of software design. ASE2007, pages 441444, November 2007. Holtta-Otto, K. 2005. Modular product platform design. Espoo: Helsinki University of Technology. 2005. Ross, A.M. 2006. Managing Unarticulated Value: Changability in MultiAttribute Tradespace Exploration. PhD Dissertation MIT. Richards, M.G. 2009. MultiAttribute Tradespace Exploration for Survivability. PhD Dissertation MIT.