Text adapted from Bots, P.W.G.

(2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

Multi-Criteria Analysis
Multi-criteria analysis is used to compare the different solution alternatives, in a logical and
consistent way, in order to make an informed decision. While the number and variety of the used
criteria can make the decision-making process more difficult, it can also make the decision-making
process richer. Collecting more criteria means collecting more information, which will cost more time
and the problem owner must set priorities. Some criteria will be more important than others. Some
criteria might be conflicting, for instance road capacity and environmental impact. The goal of low
environmental impact conflicts with the goal of a large road capacity. Alternatives that meet both
criteria are rare, therefore a balance has to be found.

Any method for finding a balanced solution must meet a number of demands. In the first place the
outcomes must be presented logically and consistently. This means that, for each of the solution
alternatives, a clear indication must be given for each criterion of what the expected impacts are if an
alternative would be implemented. In the second place, a systematic comparison of alternatives
must be possible. This means that the method for finding a balanced solution must clarify how the
various good or bad alternatives perform in relation to each other. Finally, the problem owner must
acknowledge the consequences of the set priorities. This means that the method must quickly show
that shifts in priorities lead to differences in the preferred order.

The methods for finding a balanced solution which are explained here are the Score Card and the
SMART method. These do not, by any means, cover the entire range of decision-making methods.
These are chosen with an eye to their relatively simple structure and usability in calculation.

Impact Table
All multi-criteria methods are based on the so-called impact table. An impact table is a matrix in
which each column corresponds with a solution alternative and each row with a criterion. Use the
criteria that were chosen in the goal analysis with well-defined units for the list of criteria. Also, use
the alternatives that were defined in the causal diagram and problem diagram for the list of
alternatives. The impact table is based on one earlier defined context scenario. Because alternatives
can score differently in different potential futures, it is vital to know if the alternatives are also robust
to different future developments. Robust alternatives are the ones that perform well in many
different potential futures.

Now you can indicate the expected effect of each of the alternatives per criterion. These effects can
be found in literature, by doing experiments, or making estimates. Where possible, list a value that
has a relevant characteristic. In this way, you can express the costs and impacts in financial terms
with a market price. Impacts that have no market price must be given other unique units of
measurement. If no quantification is possible, naming the expected consequences will do. You can
also use a qualitative indication of the consequences.

Example
Table 1 gives an example of an impact table with five criteria and three alternatives. The alternatives
are three ways to increase mobility in the east-west direction of a fictitious city centre area.

1
Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

A1: Reduce A2 : East-West A3 : East-West

A0: Zero Option Private Metro Service Metro Service
Transportation [10 stops] [8 stops]

C1: Construction Costs

0 50 * 106 800 * 106 600 * 106
[Euro]
C2: Maintenance Costs
800 * 103 100 * 103 600 * 103 550 * 103
[Euro/year]
C3: Transport Capacity [+-] 0 + ++ ++
C4: Transport Costs [%
with regard to the 0 10% 30% 25%
present]
C5: Travel Time [min] 31 25 16 14
Table 1: Impact Table

The three alternatives which are defined in the causal diagram and problem diagram are compared
with the Zero Option, where the present situation does not change. Alternative 1 consists of the
reduction of private transportation in the city centre, creating an area that will be used for public
transportation. Alternative 2 consists of the construction of an east-west metro service with 10 stops
to improve the public transportation system. Alternative 3 consists of the construction of the same
line with 8 stops. The criteria are obtained from the goal analysis. The data that is used for this
example is fictitious. However, the underlying causal connections will be explained.

The construction costs for the three alternatives are estimated by the different construction parties.
To reduce the amount of private transportation in the city centre and to increase the amount of
public transportation an investment of 50 million euro is needed. To construct the metro line, an
investment between 0.6 and 0.8 billion euro is needed. The expected maintenance costs of the
present infrastructure are relatively high compared to the other alternatives. The old public transport
system is outdated and needs to be upgraded to cope with the increasing demands. The
implementation of Alternative 1 also includes the upgrade of the present system. It is expected that
the number of passengers who use public transport will increase over the years. The Zero Option will
reduce the amount of available places in the system. The transport costs are expected to increase
when one of the alternatives is implemented. The travel time per alternative is based on a model of
the system.

Score Card
The score card method is used to make the comparison of different alternatives easier without
passing judgement on the alternatives themselves. The main idea behind the method is that the
solution alternatives are compared on the criteria. The emphasis is on the actual comparison. It is not
the point whether an alternative is good or bad, but to present the different alternatives in such a
way that the comparison is encouraged. This is done by setting up a score card, which gives a ranking
of alternatives per criterion, using a graphical representation. Although a score card is often used
with quantitative data, the method for its graphical representation, gradations in colours instead of
numbers, is also very useful for problems concerning bounded systems for which only qualitative

2
Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

information can be collected.

Objective and function

The objective of a score card is to establish a value for each of the criteria in one graphical overview
of all the alternatives. In this way a useful comparison of the alternatives is possible. The criteria can
vary greatly and their values will often be expressed in different units of measurement, such as
euros, metres, minutes, hectares and percentages. This means that it is impossible to simply add up
and take the highest value. The score card allows for a different way to compare the different
alternatives. The function of a score card is to support the decision maker(s) in choosing an
alternative from a collection of possible alternatives. The use of colour or greyscale ensures that the
discussion focuses on the relative relations of score per criteria, and not on the scores themselves.

Example
Table 2 gives an example of a score card with the same criteria and alternatives. The score card is
nothing more than a visual representation of the impact table. Per criterion, you indicate the ranking
of alternatives using different colours or greyscale. You colour in the score card, row by row per
criterion, and each cell in a row corresponds to the colours of the rank number of the alternatives.
Although the colours are now more important than the criterion scores, the latter must still remain
visible because the information on the expected impacts must not be lost. However, the colours and
their relative meaning are the most important thing on a score card.

A1: Reduce A2 : East-West A3 : East-West

A0: Null
Private Metro Service Metro Service
Alternative
Transportation [10 stops] [8 stops]

C1: Construction Costs

0 50 * 106 800 * 106 600 * 106
[Euro]
C2: Maintenance Costs
800 * 103 100 * 103 600 * 103 550 * 103
[Euro/year]
C3: Transport Capacity 0 + ++ ++
C4: Transport Costs [%
with regard to the 0 10% 30% 25%
present]
C5: Travel Time [min] 31 25 16 14

Ranking [lighter is better]:

Table 2: Score Card

You will have to explain your score card in terms of what the criteria are, in which units of
measurement the scores are expressed, and how you obtained a certain score. This is especially
important if there is any room for doubt about the assumptions on which the ranking is based. In an
extensive study, you could set up a separate score card for each particular aspect of a problem. The
information could then be compiled into one overview. Naturally, a compilation score card has less
detailed information. It is then important that no essential data is lost.

3
Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

Consistency rules
The score card must be a table with one row for each criterion and one column for each alternative.
You must choose colours logically, for example red for bad or risky, and green for good. Whatever
you choose, you will always have to explain the colour ranking using a legend. The colour ranking
must correspond to the ranking of the absolute criterion scores. Two (almost) equal criterion scores
must have the same colour for both alternatives. For almost all decision makers, colours have their
own value. If an alternative scores worst with regard to a certain criterion, then that alternative
could still be very useful for that criterion. The assessment of the relative importance of scores, for
the diverse criteria, is up to those who are involved in the decision making process.

Usability
The usability of a score card depends on the correctness of the assumptions behind the colour
ranking. The assumptions must not only be correct, they must also be clear for the decision makers.
You must also take into account that two (contradictory) assumptions are both plausible. The
increase of visitors to the city and the corresponding clogging of the city centre, as a side-effect of
the increased capacity, can therefore be seen as both positive and negative. If one of the interested
parties assesses an effect differently, in terms of the decision to be made, then there will be a
ranking problem. In this case it is useful to set up various score cards, which correspond to the
diverse assumptions with regard to the criteria.

Conclusion
Those who want to assess the alternatives can get a quick overview of the consequences of the
alternatives for the different criteria. In situations where many people are involved in making a
decision, and where the opinions on the relative importance of criteria are divided, it is useful to
apply this method. Those who must make a choice from the alternatives can regard the fact that no
weight is given to criteria as a disadvantage of the method. Furthermore, the score card method is
less appropriate for issues in which very many aspects play a role. In this case, the amount of
information can become an obstacle. An inherent weakness of this method is that it is easy to be
misled by its graphical representation. For example, certain aspects can be emphasised incorrectly as
a result of over-extensive separation of criteria, and also by colour choice and accentuation. In
making and assessing a score card you will always have to take this into account.

SMART
The Simple Multi-Attribute Rating Technique is a modelling tool for assessing alternatives for
different criteria, in order to make a balanced decision. The SMART method resembles the score card
method in many ways: alternatives and criteria are compared and contrasted in a matrix, and a score
is given for each alternative for each criterion. The nature of the scores, and the way in which these
scores are then used to make a balanced decision are different from the score card method.

Objective and function

Just as with the score card, the objective of a SMART-model is to establish the assessment of each
alternative for each criterion in one overview, in such a way that a useful comparison of alternatives
becomes possible. The function of a SMART model is comparable to the score card, in the way it
supports the decision maker(s) in choosing one alternative from a collection of possible alternatives.

4
Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

Because the criterion scores in a SMART model are numerical, this type of model offers more
possibilities for sensitivity analysis.

Grades Impact Scores Neutral Middle Point

Grades
10 excellent 4 great effect +3 very strong increase
9 very good 3 reasonable effect +2 considerable increase
8 good 2 little effect +1 marginal increase
7 more than satisfactory 1 very little effect 0 no effect
6 satisfactory 0 no effect at all -1 marginal decrease
5 almost satisfactory -2 moderate decrease
4 unsatisfactory -3 very strong decrease
3 very unsatisfactory
2 marginal
1 very marginal

Table 3: Examples of grade scales for a SMART model

Modelling concepts
Just as with the score card method you compare different alternatives in the SMART method, on the
basis of a number of explicit assessment criteria. Again, you compare and contrast alternatives and
criteria in a matrix. The most important difference is the way in which criterion scores are created. In
the score card method ‘absolute’ values, based on research, are used as much as possible, so that the
order of priority and the colour shading can be determined directly as a result. In the SMART
method, scores are expressed as ‘grades’: numerical values that reflect how high an alternative for a
criterion scores for the decision makers. In Table 3 three examples of scales are shown that can be
used in a SMART-model.

Alternatively, when creating a SMART-model that is based on quantifiable information, you can
normalise the data to use in the assessment. Simple multiplication of raw numerical information
would give strange results if large scores are juxtaposed to small ones on other criteria in the same
table. The raw scores have to be made comparable. This can be done by defining a range and then
transforming the raw scores in the table into a score between 0 and 1. The simplest form is to reset
the largest raw score to 1 (if a large score on that criterion is desirable) or 0 (if a large score is
undesirable). The smallest raw score in the table is then reset to 0 (large score desirable) or 1 (large
score undesirable). All values in between are recalculated proportionally.

For the criterion ‘construction costs’, as seen in Table 1, a low value is what the municipality wants.
Reset the score of the alternative with the lowest score on this criterion to 1. The highest value (for
A2) in the table for this criterion is set to 0. Now the value of the other criteria, A1 and A3 are
recalculated as:

Range = best score (A0) – worst score (A3) = 0 - 800 * 106 = -800 * 106
Normalised value of construction costs for A1 =
(actual value-worst score)/(range) =
(50 * 106 - 800 * 106) / (-800 * 106) = 0.9

5
Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

It does not matter which scale you use in applying the modelling technique, as long as those who
interpret the results are informed about the used scale.

A second way in which the SMART method clearly differs from the score card method is the
possibility of assigning different weights to criteria. Ranking alternatives is then done by establishing
the weighted sum of the scores per alternative. The higher the total score, the better the alternative.
It is important to note that the SMART model is specifically designed for one actor, as everybody will
have different goals and thus ideas about what is important and what is not. Therefore, the first step
is to determine for which scenario you will make the SMART model but also for which actor. The
actor decides about the weight per criterion. Make sure you motivate why that specific weight is
given. Now you can determine the score by multiplying the weights with the score per alternative.
The sum of scores is your total score which can be compared for the different alternatives. This is
shown in Table 4.

Alternatives
Criteria Weights A1 A2 ... An
C1 G1 S11 S21 Sn1
C2 G2 S12 S22 Sn2
. . . . .
. . . . .
. . . . .
Cm Gm S1m S2m Snm
m m m
Total scores  Gi·S1i Gi·S2i ... Gi·Sni
i=1 i=1 i=1

Table 4: Calculating the weighted scores

Model construction
In constructing a SMART model, you start by indicating the alternatives and criteria in the table. Then
you choose the scale for which the alternatives per criterion have to be scored or normalized, and
you let the decision makers determine the scores or you normalise the scores based on the raw data.
Then you let them determine the weights per criterion. If there is more than one decision maker you
will have to go through a process with them, until you have reached enough of a consensus about
the scores and weights. It is useful, in any case, to analyse the individual scores and weights of the
different actors because this gives variants for calculation in the sensitivity analysis.

Example of a SMART model with grades

Table 5 gives an example of a SMART model with the subjective interpretations by the city
municipality of the anticipated impacts. It is vital to explain why the grades and weights are chosen.
The municipality is mainly concerned about the construction costs and the expected increase in
transportability of the citizens and visitors. Therefore, these criteria are weighted 3 and 4, in
comparison to the criterion transport costs.

6
 Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

A1: Reduce A2 : East-West A3 : East-West

Weight A0: Zero Option Private Metro Service Metro Service
Transportation [10 stops] [8 stops]
C1: Construction Costs 4 8 7 4 5
C2: Maintenance Costs 2 5 9 6 7
C3: Transport Capacity 3 3 6 9 8
C4: Transport Costs 1 8 7 5 6
C5: Travel Time 3 4 5 8 9
Total Weight 71 86 84 91
Ranking 4 2 3 1
Table 5: SMART model with grades and weights from the actor perspective ‘’City Municipality’’

Example of a SMART model with normalised scores

Another way of constructing a SMART model is to normalise the data that is obtained during system
analysis. This is done when the system information is quantifiable, such as in Table 6.

A1: Reduce A2 : East-West A3 : East-West

A0: Zero Option Private Metro Service Metro Service
Transportation [10 stops] [8 stops]

C1: Construction Costs

0 50 * 106 800 * 106 600 * 106
[Euro]
C2: Maintenance Costs
800 * 103 100 * 103 600 * 103 550 * 103
[Euro/year]
C3: Transport Capacity 11 * 103 12 * 103 15 * 103 14 * 103
C4: Transport Costs [%
with regard to the 11 15 22 20
present]
C5: Travel Time [min] 31 25 16 14
Table 6: Impact table with quantitative data

Now you can normalise the data by using following the steps:
 Determine the best score and worst score on each row. Subtract the worst score from the best
score to calculate the range
 Calculate the difference between the to be normalised values and lowest value
 Divide this by the range

Doing so will give you Table 7.

7
 Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

A1: Reduce A2 : East-West A3 : East-West

A0: Null
Weight Private Metro Service Metro Service
Alternative
Transportation [10 stops] [8 stops]
C1: Construction Costs 1 1.0 0.9 0.0 0.3
C2: Maintenance Costs 1 0.0 1.0 0.3 0.4
C3: Transport Capacity 3 0.0 0.3 1.0 0.8
C4: Transport Costs 3 1.0 0.6 0.0 0.2
C5: Travel Time 4 0.0 0.4 0.9 1.0
Total Weight 4.0 6.2 6.9 7.7
Ranking 4 3 2 1
Table 7: SMART model with normalised data and weights from the actor perspective of ‘’Travellers organisation’’

Consistency rules
In a consistency check of a SMART model you will not only have to watch the table structure
(description of criteria in first column, weights in the second column, one column for each
alternative, numerical criterion scores), but also:
 the correct application of the scale chosen for the criterion scores: Are all scores neatly within the
range of the scale? For example, are there ‘grades’ given that are higher than 10 and lower than
1?
 the correct application of the calculation method: Is the total score indeed the weighted sum of
all the criterion scores in the relevant column?
 the correct determination of ranking: Is the ranking deduced correctly? Interpretation of scale
plays an important role here. This does not mean that a high score is necessarily ‘good’ or
‘desirable’ (although it is not advisable to choose a scale that goes against this intuitive
interpretation).

Usability
The usability of a SMART model depends, in principle, on the validity of the criterion scores and the
weights. If the decision makers have insufficient knowledge to achieve or come up with a proper
score, or ‘work towards a result’ by scoring in such a way that ‘their’ alternative turns out to be the
best one, then a useful comparison will be seriously hampered. Another cause of non-valid scores is
if the decision makers interpret the scale incorrectly. The risk is that a high value is automatically
translated into a high score in the impact table (for example high costs), whereas high costs should
be seen as disadvantageous. It is therefore advisable to check the scores together with the decision
makers again before calculating the model.

Application
A SMART model gives insight into the ranking of alternatives after the calculation of total scores.
However, the point of decision making is certainly not to simply adopt the alternative that scored
highest in the model calculation. The added value of a SMART model is in the opportunity to carry
out a ‘sensitivity analysis’. You do this by using the model to calculate different combinations of
weights and checking the degree to which small and larger changes in weights lead to changes in the
ranking of alternatives. It is particularly interesting to use the model to calculate situations with more

8
Text adapted from Bots, P.W.G. (2002), Reader: Introduction to Policy Analysis, Chapter 6, Decision Making van Kersbergen, R.J. (2016)

than one decision maker, for each separate decision makers’ ‘set of weights’. If it appears that each
different ‘set of weights’ leads to very divergent ‘best’ alternatives, then a discussion and
supplementary information will be needed in order to be able to make a decision.