World Bank Policy on Selection and Employment of Consultants

WORLD BANK POLICY ON SELECTION AND
EMPLOYMENT OF CONSULTANTS: STUDY OF ITS

EFFECTIVENESS
Giovanni Casartelli Elmar Wolfstetter

Operational Policy and Country Services A. von Humboldt University
Procurement (OPCPR) Berlin
Working Paper
(Final Draft) March 4, 2007
THE WORLD BANK

3
Contents
Acknowledgements ........................................................................................... 9
Accronyms and Abbreviations....................................................................... 10
1 Overview .................................................................................................. 12
2 Key Findings............................................................................................ 13
3 Main Recommendations......................................................................... 16
II Study Purpose and Design ............................................................22
4 Introduction............................................................................................. 22
5 The Study’s Objective, Approach and Target Audience .................... 23
6 The Study’s Design ................................................................................. 24
6.1 Characteristics of selected assignments ......................................... 24

6.2 Data Analysis Tools ....................................................................... 26
III Findings ..........................................................................................30
7 Efficiency and Economy ......................................................................... 30
8 Participation ............................................................................................ 33
9 The First Phase of the Selection Process: Preparing the Request for
Proposals (RFP) .............................................................................................. 36
9.1 Terms of Reference ........................................................................ 36

9.1.1 Grading of the ToR ........................................................... 37
9.2 Staff Months and Assignment Cost Estimates ............................... 40
9.3 Evaluation Criteria and Minimum Technical Score....................... 41
9.3.1 Setting the Evaluation Criteria.......................................... 42
9.3.2 The Minimum Technical Score......................................... 43
9.4 Choice of the Selection Method..................................................... 44
4
9.4.1 Guidelines Para. 3.2: Appropriate Selection Method for
Complex Assignments ................................................................... 45
Assignments with Mixed Short Lists ............................................. 46
9.5 Setting the Weight of Quality ........................................................ 46
9.6 Choice of the Contract Model........................................... 47
9.7 Consideration for Competition: The Short List ............................. 47
9.7.1 The Request for Expressions of Interest (EoI).................. 48
9.7.2 Homogeneous Short List with a Wide Geographic Spread
49
9.7.3 The case of “Other Organizations” ................................... 51
10 The Evaluation Committee .................................................................... 51
11 The Second Phase of the Selection Process: Proposal Evaluation...... 54
11.1 The Evaluation of Technical Proposals ......................................... 54

11.1.1 Analyzing the Award Distribution.................................... 55
11.1.2 High-and Low-Ballers ...................................................... 56
11.1.3 Using QCBS in lieu of QBS ............................................. 57
11.2 Analyzing the Score Distribution of the Sample ........................... 57
11.2.1 Detecting Disagreement.................................................... 58
11.2.2 Detecting Decisive Bias .................................................... 59
11.2.3 Tools for neutralizing outliers........................................... 60
11.2.4 The Importance of EC’ Collaboration .............................. 61
11.3 The Scoring Rule ........................................................................... 61
11.3.1 Summary ........................................................................... 61
11.3.2 Excursus: The ideal scoring rule....................................... 61
11.3.3 The currently used scoring rule......................................... 62
11.3.4 The currently used scoring rule is flawed ......................... 63
11.3.5 It favors higher priced proposals....................................... 63
11.3.6 It complicates bidding....................................................... 65
11.3.7 It unnecessarily complicates calibration ........................... 66
11.3.8 It allows “irrelevant proposals” to change the selection ... 67
5
11.3.9 The marginal reward for quality (MRQ)........................... 69
11.3.10 Are these flaws empirically significant?........................... 71
11.3.11 Conclusions....................................................................... 73
11.4 The Contract................................................................................... 74
12 National Consultants Participation ....................................................... 75
12.1 Part II Consultants participation rates ............................................ 76

12.2 Creation of National Consultancies as Development Objective .... 78
13 Transparency........................................................................................... 79
13.1 Clarity and Comprehensiveness of Guidelines and RFP ............... 80

13.2 Regulating Evaluation and Selection ............................................ 81
13.3 Commitment of EC, Client and Consultants.................................. 81
14 Warning Signs ......................................................................................... 81
IV Recommendations .........................................................................84
15 Raise Efficiency and Economy............................................................... 85
15.1 Bundel Assignments and Allow Continuations ............................. 85

15.2 Rationalize Recourse to Request for Expressions of Interest ........ 85
15.3 Encourage the Use of Available Simple Selection Methods and
Simplified Procedures............................................................................... 87
16 Raise Consideration for Quality ............................................................ 87
16.1 Introducing the Consideration for Risk.......................................... 87

16.2 Avoiding Misapprehension: Rephrasing the Guidelines................ 88
16.2.1 Para. 1.5 (General Considerations) ................................... 88
16.2.2 Para. 2.1 (Adoption of QCBS) .......................................... 88
16.2.3 Para. 2.16 (Evaluation Criteria) ........................................ 89
16.2.4 Paras. 2.16 and 2.17 (Evaluation Criteria “Methodology”
and “Key Personnel”)..................................................................... 89
16.2.5 Para. 2.16 (Evaluation Criterion Transfer of Knowledge) 90
6
16.2.6 Para. 2.23 (Combined Quality and Cost Evaluation) and
para 2.11 (Instructions to Consultants) .......................................... 91
16.2.7 Para. 3.2 (Adoption of QBS)............................................. 91
16.2.8 Para. 3.5 (Adoption of FBS) ............................................. 92
16.3 Reduce Burden of RFP Preparation and Proposal Evaluation....... 92
16.4 Composition of the Shortlist .......................................................... 93
17 Composition and Operation of the Evaluation Committee................. 94
17.1 Composition of the Evaluation Committee.................................... 94

17.2 Operation of the Evaluation Committee ........................................ 95
17.3 Adding an Independent Eye and Voice.......................................... 95
18 Improving Evaluations - Weeding out Bias, Corruption, and Shill

Bidding ............................................................................................................. 96
18.1 Improving Evaluations ................................................................... 96

18.2 Weeding out Bias and Corruption.................................................. 97
18.2.1 Use “purified” scores ........................................................ 97
18.2.2 Use either “median” or “trimmed mean” aggregation ...... 97
18.3 Weeding out “shill bidding” .......................................................... 98
18.4 Improving the Scoring Rule in QCBS ........................................... 99
19 Encouraging the Development of National Consultants: Local

Knowledge becomes Key .............................................................................. 100
19.1 Consider the Need for Stronger Guidance ................................... 100

19.2 Short Term Recommendations..................................................... 101
19.3 Long Term Recommendations; Developing a Vision and a Policy102
20 Enhancing Transparency of Evaluations............................................ 103
21 Tools for Managing and Monitoring Selections................................. 104
V Annexes.........................................................................................106
Annex 1: Assignment’s Terms of Reference ........................................... 106
Annex 2: Sample of One Project Sheet.................................................... 111

7
Annex 3: Purifying Technical Scores....................................................... 116
Annex 4: A Monte-Carlo simulation of median aggregation ................ 118
Annex 5: Decisive bias of coalitions of evaluators .................................. 120
Annex 6 Improving the Scoring Rule in QCBS...................................... 122
1 Summary ...................................................................................... 122

2 The proposed new scoring formula.............................................. 122
3 Why this formula?........................................................................ 124
4 Another comparison of the “old” and “new” scoring rule ........... 126
5 Generalization to “diminishing returns to quality”? .................... 127
6 Calibration: “No one–size–fits–all” ............................................. 128
7 Beware of the limits of relative scoring ....................................... 129
Annex 7: Historical Development of Bank Guidelines........................... 130
Annex 8: Selection Rules of Other Agencies ........................................... 133
Annex 9: Example of Project Procurement Plan ................................... 137
Bibliography .................................................................................................. 139

8
9
Acknowledgements
This study has been prepared by a team headed by Giovanni Casartelli,
(Consulting Services Adviser OPCPR) and Professor Dr. Elmar Wolfstetter
(Consultant OPCPR)1 assisted by Martin Ehrenberg and Anders Pettersson.
Franco de Siervo and Piero Ravetta (Consultants) read and commented
extensively upon different drafts. Peer reviewers were Alfonso Sanchez, Renko
Campen, Eigil Pedersen, Enrique Pinilla, Jaime Roman (Consultants, OPCPR),
Els Hinderdael (Manager SARPS), Patricia Macgowan (Sr. Procurement
Specialist LCSPT), Jerry Lebo (Country Program Coordinator, EACNF),
David Potten (Head, Trust Fund Program Administration, TFO) and Joel
Turkewitz (Sr. Procurement Specialist, SARPS). Jan Walliser (Economic
Adviser, OPCCE), Eduardo Talero (Consultant) and Felix Prieto
(Sr.Procurement Specialist LCSPT), provided insightful comments. Carla
Bertoncino (Sr.Economist AFTH1) and Prof. Hina Nazar have lent themselves
to the interpretive process that the study has required. While former OPCPR
Manager Robert Hunja suggested, encouraged and supported the preparation of
the study, Bernard Becq (Chief Procurement Policy Officer, OPCPR)
supported its finalization and provided insightful comments. Patricia Rogers
edited the final draft. Most of the above mentioned people have not only
reviewed this draft but for years have participated in the discussion that has led
to the study itself. About 60 Bank staff, Task Team Leaders and Procurement
Staff provided observations and feedbacks along with professional and expert
consultants in different positions and sectors.
1
Dept. of Economics, Humboldt University at Berlin, Spandauer Str. 1, 10099 Berlin,
Germany
10
Accronyms and Abbreviations

ADB – Asian Development Bank
CQS – Selection Based on Consultants’ Qualifications
CSM – Consultant Services Manual
EA – Executing Agency
EC – Evaluation Committee
EoI – Expressions of Interest
FBS – Fixed Budget Selection
FTP – Full Technical Proposal
GPN – General Procurement Notice
LCS – Least Cost Selection
MTS – Minimum Technical Score
NGO – Non Government Organization
ODA – Overseas Development Assistance
OPCPR – Operations Policy and Country Services Procurement
PAD – Project Appraisal Document
PS – Procurement Specialist
QBS – Quality Based Selection
QCBS – Quality and Cost Based Selection
RFP – Request for Proposals
SPN – Specific Procurement Notice
SRFP – Standard Request for Proposals
SSS – Single Source Selection
STP – Simplified Technical Proposal
ToR – Terms of Reference
TTL – Task Team Leader
11
Executive Summary
12
Executive Summary
1 Overview
1. Effective selection and use of consultants is necessary if development
projects are to achieve their intended impact. The Guidelines: Selection and
Employment of Consultants by World Bank Borrowers (the Guidelines) are
intended to lead borrowers to secure consultants’ services by efficient contract
allocation (that is, by means that return value for money). This study
investigates how effectively the 1997 and subsequent editions of the Guidelines
are in achieving this goal by meeting the considerations of quality, efficiency,
economy, competitive fairness, transparency, and development of national
consultants.
2. Despite the Bank’s efforts to improve its assistance, this study finds that
borrowers are not getting the best out of their procurement of consulting
services.
i) The Guidelines considerations are not being met—particularly with

respect to quality, efficiency, and economy—and often the results
obtained are not those intended.
ii) The Guidelines contain obscurities, contradictions, and
imperfections that confuse borrowers, consultants, and Bank staff
who try to interpret the provisions and apply sound judgment. This
problem is due at least in part to the way in which the Guidelines
have been amended over about 40 years to reflect new and better
practices or to factor in changing Bank agendas (such as
transparency and anticorruption initiatives).
iii) Compliance with the Bank’s many rules is becoming an
independent objective to which efficient contract award is
sacrificed. In making procurement decisions inexperienced
borrowers often fail to take into account the complexity and the risk
profile of the consulting assignment, and confuse the priorities that
should dominate the selection process.
iv) Bank staff often fails to enforce the Guidelines, especially when
they accept without question whatever the borrower may propose on
selection methods and other important decisions, regardless of the
complexity of the assignment and value of the project.
(Unfortunately, staff themselves may not have sufficient knowledge
to advice borrowers.)
v) Given these problems, the best consultants often abstain from
participating and others complain of a “race to the bottom” that
13
makes consulting frustrating as a profession and unsustainable as a
business proposition.
3. In light of these findings, this report recommends amendments designed

to reestablish the effectiveness of the Bank Guidelines and of their related
selection mechanisms. However, the authors suggest that proposing yet more
amendments to the existing Guidelines may not be the most effective approach.
The increasing variety, complexity and specialization of consultant services
bring about a widening asymmetry of information between clients and
consultants, which the present Guidelines do not help borrowers cope with.
Thus, it has become more and more evident to the authors as this study was
approaching completion that this report affords an opportunity that should not
be wasted to trigger a more radical and forward-looking reform of the
Guidelines.
4. It is important to add that, while changing the Guidelines will help

address the deficiencies of the selection process, it will also be necessary to
address the problem of incompetent and unethical service delivery by
consultants, and incompetent or unethical management by borrowers. As in so
many other areas of aid effectiveness, building capacity in effective and ethical
practices—on both the selection/management side and the service delivery
side—is essential.2
2 Key Findings
5. Para. 1.4 of the Guidelines sets out the considerations that should
govern the selection and employment of consultants. For the orderly
identification and evaluation of issues, the study follows the Guidelines’
presentation of those considerations in the same sequence. The study’s findings
are based on the data obtained from a random sample of 120 Bank-funded
consulting contacts awarded by borrowers in FY03-04.
6. Quality. The complexity and specialization of consultant services keep

increasing, but borrowers disregard these factors when deciding the selection
process parameters and the shortlist. They give little consideration to the risks
associated with poor selections. Although the Guidelines offer six different
selection methods for use in all possible situations, borrowers choose the same
method (quality- and cost-based selection, or QCBS) in 92 percent of cases—
and disregard the characteristics of the consulting assignment and the
composition of the shortlist, even when doing so can put at risk the quality of
the services and the project itself. This suggests that borrowers lack either the
2
OPCPR is carrying out two related studies, one on “Integrity and Corruption in Professional
and Expert Services”, and the other on the Role of Consultants in Clients’ Capacity
Development.
14
expertise or the guidance to make this choice. As a result, the request for
proposals (RFP) functions as an imprecise discovery mechanism that
eventually discourages good consultants from competing: in only 57 percent of
cases do the invited consultants submit technical proposals that satisfy the RFP
minimum technical score (MTS). Interviews with staff and file correspondence
suggest that the Guidelines’ confusing language on the use of the different
parameters, including the selection methods, contributes to the
misinterpretation of the Guidelines by borrowers, and possibly also by Bank
staff.
7. Borrowers’ competence in preparing Terms of Reference (ToR), in

shortlisting consultants and evaluating proposals is being stretched as the
complexity of the services to be procured increases. In the study sample, only
38 percent of the EC members were specialists in the disciplines of the
assignment and the rest were executing agency directors and delegates of
higher authorities not always with the necessary experience. Evaluation
committee (EC) members exercise considerable discretion in determining the
contract award that may have little to do with the consultant’s qualifications.
Problems of professional behavior on both consultant and borrower sides point
to the critical need to establish effective means of obtaining better and timely
evaluations. 131 days on average elapse between the proposal submissions and
the transmission of the evaluation reports to the Bank.
8. Too much attention is paid to price competition and too little to the
quality of services, the project’s or the borrowers’ needs. EC members assign
similar quality scores to technical proposals that are quite differently priced,
and the sample cases show no significant correlation between quality and price
(ρ 0.09). In the sample, the average dispersion coefficient of the technical
scores is 1.12, while the price dispersion is 1.79. In 67 percent of the cases the
lowest-cost proposal is awarded the contract, and in 37 percent the proposal
with the highest technical score is also the lowest priced. Conversely, the
highest technical score is awarded the contract in 64 percent of cases. For the
study sample’s highest subset of risky and complex assignments, the technical
dispersion increases only minimally to 1.13, while the price dispersion jumps
to 2.10 and the lowest bid gets the award in 80 percent of cases.
9. The scoring formula for QCBS that the Guidelines prescribe (2.22) has
undesirable characteristics: (a) it favors higher-priced proposals; (b) it depends
on the lowest price in a way that makes it difficult for consultants to predict
how the rule rewards them for quality; (c) it makes it difficult for the borrower
to calibrate the scoring rule to the specifics of the selection case because it
includes the minimum price; and (d) it allows irrelevant proposals to change
the selection in a way that invites shill bidding. These flaws would not be so
troublesome if proposals prices were closer to each other. But in fact prices
show a high dispersion: in 28 percent of all QCBS cases in the sample, the
15
price dispersion factor is above 2. Even if ECs evaluate quality correctly, and
impartially, the scoring formula would distort the results. Moreover, the study
sample shows that the correlation between technical and financial scores (i.e.
between quality and price) is neither positive nor negative.
10. Efficiency and Economy. In most cases the selection process is taking
an unreasonably long time and is too expensive. On average the selection
requires 17 months and 400 to 700 pages of technical information, even though
the median value of the contracted assignments is only US$432,000. Preparing
the proposal is also costly: it can cost $30,000 to $60,000 to prepare a proposal,
with one chance in five of success. Yet the information value of proposals can
be low: the long delays in the award process mean that work plans and CVs
must very often be changed before the assignment starts. Good consultants
soon lose their appetite for participating when they consider the costs involved
and the slim chances of adequate reward, so World Bank-funded projects do
not always get the benefit of quality services. For lack of explanation in the
Guidelines, the consideration of “economy” of the selection process is often
confused by borrowers and Bank staff with the cost of the consulting services.
11. Fair Competition. Borrowers generally comply with the requirements

for a shortlist of six consultants with “wide geographic spread,” with “not more
than two” from the same country but “at least one from a developing country.”
However, this compliance can mean overlooking such important requirements
as participants’ qualifications, fairness, commensurability, client preferences,
and the integrity of the selection process. The presence of organizations “other
than consultants” in the shortlist together with independent, private consulting
firms, is an additional source of unfairness in spite of the Guidelines’ explicit
instructions (para 2.8) towards a balanced short list.
12. Development of National Consultancies. Bank statistics and the study

sample suggest that the Guidelines’ provisions promoting national consultancy
and the adoption of QCBS contribute to increasing national consultants’
participation in Bank-funded assignments. However the consulting industry in
borrowing countries is mostly inadequate and failing, plagued by ineffective
legislation, corruption and low erratic demand. Widespread public and political
indifference, inadequate regulation with too-frequent recourse to price-based
selection methods, and disregard for borrower needs, lead to low remuneration
levels; these compromise the quality of services, the independence and the
survival of domestic consultants that depend on work from the public sector
orders. The departure of valuable professionals—to other activities or other
countries—is one of the most painful consequences. The Bank should consider
replacing the Guidelines’ present preferential provisions that aim at
encouraging local participation quantitatively with measures that attract and
more vigorously promote the quality and sustainability of local consulting
firms.
16
13. Transparency. The consideration of transparency was patched into the

Guidelines in January 1999, although the Guidelines had not originally been
written with transparency as one of the explicit considerations. The provisions
related to transparency are inspired by the one-step procurement of goods and
works, in which the numerous bids cost less to prepare and are easier to tamper
with. The result of this approach was an increased burden on Bank and
borrowers in the form of added controls and reviews. The approach overlooks
the fact that, given the high cost of preparing proposals consultants have an
interest in complying with formal requirements to avoid being disqualified for
a minor non-compliance, and that the borrower wishes to retain as many as
possible of the few proposals being submitted. More important, the Guidelines
do not consider the obscuring effect of a puzzling scoring formula (the QCBS
formula) that leads to unclear technical evaluations, nor do they mention the
transparency required in appointing EC members, conducting evaluations, and
discussing and agreeing among EC members before an award.
14. Conclusion. The outstanding conclusion of the study is that rigid or

uncritical compliance with the present specification-driven selection process
becomes increasingly ineffective in terms of efficient contract award, as
assignments move from simple to complex and the consulting services grow
more specialized. The most frequently used selection method, with its related
scoring rule, is not always the most suitable and should be used more
prudently: its emphasis on cost saving induces consultants to bid low,
discourages quality, and ignores project risks. Instead of solving the selection
problem efficiently, the introduction of price as a decision factor in the manner
described, offers a shortcut that misleads borrowers when proposals are not
really comparable. This study confirms that while less-than-desirable
procurement results can sometimes be attributed to the borrower’s lack of
experience and to corruption, they often stem from complicated, imprecise
procurement tools that can themselves lead to corruption.
3 Main Recommendations
15. This study’s recommendations focus on those provisions of the
Guidelines that are incorrect or confusing and as a consequence leading to
wrong decisions or poor applications. The Guidelines should not be left as
they presently are. Two options are available: to revise the present Guidelines
or to rewrite them. The study’s conclusions suggest that the present version is
not working as intended also because it has been patched over the years when
new important considerations had to be added, resulting in a document that is
not entirely clear and coherent. However, the following eight key
recommendations propose only a revision.
17
Raise Efficiency and Economy
16. Key Recommendation 1: Reduce complexity. This can be achieved

under the present guidelines by: (a) bundling compatible assignments whenever
technical and administrative circumstances allow, or permitting the
continuation of sequential assignments if the incumbent has performed well;
(b) unifying multiple requests for expression of interest under the same project,
and encouraging consultants to be selective in expressing their interest; and (c)
encouraging borrowers to use simple selection methods such as “selection
based on the consultant’s qualifications” (CQS), and of the “simplified
technical proposal” (STP) option that is already offered in the harmonized
Standard RFP.
Increase Quality
17. Key Recommendation 2: Clarify and simplify the guidance. This can
be achieved by (a) rephrasing the Guidelines to avoid indicating that one
selection method is “recommended,” and link the choice of selection method to
the assignment terms of reference (TOR) and the value of the project; (b)
clearly describing the conditions for the use of each selection method; (c)
reducing the complexity of RFP calibration and proposal evaluation by
adopting STP and CQS for set dollar amounts (along the lines already adopted
by the Asian Development Bank). These measures alone, without altering the
present policy, can improve application of the Guidelines, reduce the overload
of technical information, and establish the importance of consultants’ past and
verifiable qualifications for a more reliable selection.
18. Key Recommendation 3: Stress the importance of the “qualifications

for the assignment” and of “fair competition” when comprising the short list.
This can be achieved by (a) relaxing the requirement of “six firms with wide
geographic spread” (2.6), and (b) formulating more clearly the concept of a
homogeneous (as against mixed) shortlist (2.8). These clarifications would also
help borrowers choosing more correctly among the available selection
methods.
19. Key Recommendation 4: Affirm the importance of a professionally

qualified EC. A provision affirming the importance of appointing a trusted EC
composed of borrower staff that can evaluate proposals knowledgably and
impartially will contribute to the quality and credibility of the selection
process. When needed, the Guidelines should allow for or require the presence
of an independent observer “without vote or voice” appointed by the borrower
to vouch for the impartiality of the evaluation. This provision would encourage
more qualified consultants to participate.
18
20. Key Recommendation 5: Reconsider the QCBS scoring formula. The
present QCBS scoring formula can distort an evaluation, particularly when the
dispersion of proposal prices is high; alternative scoring formulas were
examined but did not yield significantly better results. A new formula
developed in the course of this study, without the defects of the present one,
should be piloted. In the meantime, it is recommended that the existing QCBS
formula be used only, when justified by the TOR and the project’s features, the
consultant shortlist is homogeneous, and technical proposals are expected to be
comparable; in such cases price constitutes an efficient and transparent
selection criterion. In other situations, making a trade-off between quality and
price may yield unfavorable results and expose the underlying project to
unwanted risks.
Encourage Development of National Consultants
21. Key Recommendation 6: Promote development of the national

professional and expert consultants. This domestic policy consideration could
be improved by (a) increasing the contract value thresholds for national
competition; and (b) using CQS, quality-based selection (QBS), and fixed-
budget selection (FSB) as default methods, resorting to QCBS only when that
method is justified by the characteristics of the assignment and the shortlist.
This approach will encourage national consultants to build their professional
qualifications and focus on the quality of services, leaving price competition
for the relatively few cases in which it is justified. If emulated by governments
(and donors), this approach could help revitalize depressed domestic markets,
slow the departure of national professionals, and attract international
consultants to establish activities in these countries as a form of FDI in
professional capital. Furthermore the Bank could abstain from any particular
preferential Guidelines provisions, leaving to borrowers the task of developing
policies and regulations for the benefit of their struggling consulting industries.
Increase Transparency
22. Key Recommendation 7: Improve Guidelines treatment of

transparency. The selection process would benefit if the Guidelines explained
how transparency can contribute to an efficient contract award. Clarifying the
text of the Guidelines on crucial questions and decision points will increase the
transparency of the selection process for all parties involved. In addition to
transparency, the Guidelines should mention “ethical performance” in setting
out ethical performance parameters for consultants and borrowers.
Transparency and ethical performance would offer a rational basis for building
trust and cooperation between borrowers and consultants.
19
Final Consideration
23. As the complexity of projects increases, the specialization and

complexity of the consulting services that borrowers need to procure are also
bound to increase. This evolution inevitably generates asymmetries of
information and principal-agent problems that are often at the root of
incompetent or dishonest procurement practices. The prevailing procurement
culture attempts to address imperfect technical evaluations and unethical
decisions with increasingly rigid and complicated requirements and with
sanctions that ultimately defeat the efficiency of the guidance and the
effectiveness of its outcomes. The remedy to this distressing state of affairs is
most likely a procurement approach that considers overcoming information
gaps not so much by costly control systems but through a combination of
compliance and mutual (rational) trust or partnership between borrowers and
their consultants. The Bank is conducting a study on integrity in the selection
and use of consultants that is expected to provide important elements in this
direction.
21
Part II
Study Purpose and Design

22
II Study Purpose and Design

4 Introduction
Bank financed projects require preparation, implementation supervision and
technical assistance services that, under Clients’ responsibility, are entrusted
for the most part to consultants selected and hired by Borrowers and funded by
the Bank.3
Although consultants’ services have evolved to accommodate changing

projects and circumstances, it is often argued in the procurement debate that, in
spite of this evolution, the Bank has fallen behind best practices with its policy
on the selection and use of consultants. It is also commented that the present
Bank’s Guidelines need to be better tailored to the changing Bank portfolio,
and that the manner in which clients and Bank staff interpret and comply with
them is too rigid and far from efficient. This increasingly discourages quality,
innovation and good consultants from seeking involvement in Bank projects.
Specific concerns from Bank clients, Bank operational staff, executive

directors and the consultants associations refer to the capability of the selection
process specified in the Guidelines actually leading to the identification of the
most suitable consultant for the assignment at hand and to efficient contract
allocation. At the 2005 BIMILACI, the Biannual Meeting between World
Bank, Inter-American Development Bank, other multilateral institutions and
the International Federation of Consulting Engineers, the European Federation
of Consulting Associations and other umbrella consultants’ associations,
OPCPR offered to conduct a study on the effectiveness of Bank policy
guidance on the selection and use of consultants by its borrowers.
A draft of the study terms of reference (Annex 1) was circulated among the
various stakeholders. The study executed by OPCPR, started in November
2005, was made possible by contributions from Bank Consultants Trust Funds
of the Governments of Denmark, Germany, Italy, The Netherlands and
Sweden.
3
In 2004 the Bank has funded consulting services contracts for about $ 1.3 billion.
23
5 The Study’s Objective, Approach and Target Audience
The purpose of the “Guidelines on Selection and Employment of Consultants
by World Bank Borrowers”, 2004 Edition, the Guidelines is to describe Bank’s
policies and procedures to be followed for the efficient allocation and
execution of consulting contracts in Bank funded projects.
This is to be achieved by applying the Guidelines’ provisions keeping in the

forefront their main considerations:
• quality,
• efficiency and economy,
• fair competition among consultants,
• participation of national consultants from development countries, and
• transparency.
An efficient contract award, i.e. appointment of the most suitable consultant,

requires striking the best balance among each of the above considerations
(Guidelines para 1.4)4 and by correctly applying the procedures and parameters
that can effectively dispose of market and institutional failures emerging in the
specific selection case.
The key assumption of this study is that efficient contract allocation and the
satisfaction level of the interested parties can be explained to quite a good
extent by the way selection is regulated, the degree to which clients are able to
apply the regulation, create genuine competitive situations, and from the results
of the selection.
The objective of this study is first to assess to what extent the present
Guidelines lead clients achieve efficient consultant selection (value for money)
by addressing the above policy considerations and what may prevent them
from doing so. Second, it is to propose specific clarifications, updates and some
changes that would make the guidance more understandable and user-friendly
(transparent). Third, it is to suggest how the Guidelines could be modernized in
accordance to changing procurement best practices on selection and
employment, evolving Bank portfolio, and market as well as institutional
conditions faced by clients and consultants.
The study approach consists in investigating the selection process from the
particular angle of each of the above considerations. Consistent with the
procedures specified in the Guidelines, each step of the selection process is
examined in its measurable and significant dimensions, from the publication of
the initial request for Expressions of Interest (EoI), to the contract signature.
4
Neither the Guidelines nor the Bank SRFP provide definitions of the above “considerations”.
24
Obviously the problem extends beyond procurement regulation purely, as
problems of incompetent and unethical service delivery by consultants, or
incompetent or unethical client procurement decisions also pose risks that need
to be addressed as they can directly affect the procurement process and its
outcome. These aspects were always considered when observing facts and
recommending solutions but where not the explicit object of this study.
The target audiences are the Bank and the above mentioned participants to the
May 2005 BIMILACI. They pointed to the need to deepen the dialogue on
service quality and on the sustainability or effectiveness of current Bank
policies regulated by the present Guidelines. Bank clients, its operational staff
and the consultants may also be interested in this study.
6 The Study’s Design
6.1 Characteristics of selected assignments
The study is based on information covering Bank funded, client executed

consulting assignments awarded during FY 03/04 obtained from Bank files,
IRIS, Bank field offices, interviews with Task Team Leaders (TTL),
procurement specialists (PS) and with consultants.
120 consulting services contracts were randomly selected from a population of

1972 prior review contracts, from Bank operational projects proportional to
Bank lending in its six operational regions in FY03/04.
Eventually, of the 120, only 114 selection processes leading to assignment

contracts were studied in detail since it was impossible to obtain complete data
for the remaining 6.5
5
The data collection phase took place from mid of November 2005 until mid of April 2006.
25
Table 1: Study Sample by Sector and Region
Sectors/ Regions ECA SAR EAP MENA LAC AFR Total
Agriculture 3 2 0 0 2 3 10
Education 2 1 1 2 1 1 8
Transport 2 6 2 0 2 8 20
Energy 4 1 1 0 0 4 10
Water 2 1 2 0 3 4 12
Health 2 7 2 2 1 4 18
Social 1 2 0 1 1 1 6
Public Admin & Law 4 2 1 0 4 6 17
Finance 2 3 1 0 4 1 11
Industry & Trade 1 0 1 0 0 0 2
Total 23 25 11 5 18 32 114
The choice of FY 03/04 as the study years was dictated by the availability of
recent information in Bank files and testimony from Bank staff. The backdrop
of this choice is that not all assignments examined have been completed yet.
For the purpose of this study the complete process of selection is split into two
phases, i) preparation of the RFP, including the short list, and ii) proposal
evaluation. Figure 1 describing the selection process for the different selection
methods contemplated by the Guidelines should facilitate reading the text.
26
Figure 1: Steps of the Selection Process
Finalize T O R
Finalize Cost Estimate

and Budget < USD 100,000 < USD 200,000
QBS
QCBS SSS LCS CQS
FBS
Call for Call for Call for

Expressions of Interest Expressions of Interest Expressions of Interest
Prepare Shortlist Prepare Shortlist Prepare Shortlist Prepare Shortlist
Define Eval. Criteria & Define Eval. Criteria & Define Eval. Criteria &
Min. Qualif. Mark Min. Qualif. Mark Min. Qualif. Mark
Prepare RFP Prepare RFP Prepare RFP Prepare RFP
Bank's No-Objection Bank's No-Objection Bank's No-Objection Bank's No-Objection
Send RFP to the Selected

Send RFP Send RFP Send RFP Send RFP
Consultant
Prepare & Submit Tech. & Prepare & Submit Tech. & Prepare & Submit Prepare & Submit Tech. & Prepare & Submit Tech. &
Finan. Proposals Finan. Proposals Technical Proposal Finan. Proposals Finan. Proposals
Evaluate Evaluate Evaluate

Technical Proposals Technical Proposals Technical Proposals
Bank's No-Objection Bank's No-Objection Bank's No-Objection
Public Opening of Finan.

Public Opening of Finan. Public Opening of Finan.
Proposals; Calculate
Proposals if Score of Tech. FBS Proposals if Score of Tech.
QBS Corrected Price; Reject
Proposals > MTS Proposals > MTS
Proposals Whose Corrected
Perform Combined Eval. Calculate Corrected Price
Price Exceeds Budget
Negotiate with Highest Negotiate with Highest Submit Financial Proposal Negotiate with Lowest
Negotiate
Scoring Proposal Ranking Tech. Proposal & Negotiate Corrected Price Proposal
Bank's No-Objection Bank's No-Objection Bank's No-Objection Bank's No-Objection Bank's No-Objection
Contract Award
Start the Assignment

Notes: MTS = Minimum Technical Score
6.2 Data Analysis Tools
The data tools were set up in such a way that one has access to:
• A record of all relevant details of the sample assignments;
• An in-depth analysis of the relevant data of each single assignment;
• An analysis of the statistical distribution of relevant characteristics of the
population of selected assignments and its relevant subsets.
For this purpose, an Excel data sheet was developed in which all relevant
details of each of the 114 assignments were entered and then automatically
analyzed. The information on each single assignment is stored in individual
data sheets which can be opened and checked independently from other data
sheets. The following Screenshots 1 and 2 illustrate parts of the Excel data
sheets, while Annex 2 provides a complete printout of one completed form.
27
Screenshots 1 and 2: Excel Data Sheet
28
All 114 data sheets were then integrated in a large Excel Workbook that
allowed extracting the characteristics from the complete study sample for
statistical analysis. Supplementary programming was done in “Visual Basic”.
Since the study’s objective is to examine each single step leading to the final
selection of the consultant for each assignment, the data sheets provide
information on the complete selection process. In addition, various tests were
automatically carried out, from checking the duration of the selection process
or counterfactual examinations on how the outcome would have been if various
meaningful variations of the current selection rules were applied.
In principle, the data tools developed for this study could also be regularly
applied by TTL or PS who need to monitor the selection process and to keep a
close eye on its statistics.
29
Part III
Findings
30
III Findings
As explained in the introduction, for the purpose of this study, the process of
consultants’ selection is assessed on the basis of the Guidelines’ six main
considerations. The selection process is split into two phases, preparation of the
RFP including the short list, and proposal evaluation.
7 Efficiency and Economy

Lost time is never found again.
Benjamin Franklin
The duration of the selection is an indicator of the technical and administrative

efficiency of the process, and of the cost that this entails for the parties
involved. The selection of consultants currently takes far too long. Based on 97
sample assignments (the remaining 17 were single-sourced) the average
duration for selecting a consultant and completing the contract is about 17
months (Figure 2) and the median is 11 months, compared to an average 8.5
months estimated by the Bank Consulting Services Manual (CSM)6. The mean
contract value at signature is $891,000 and the median $432,0007. The average
contract duration is 26.5 calendar months. The sum of documents to be
prepared, read and processed by the Client, including evaluation of EoI’s, RFP
and proposals varies between 400 and 700 pages, independent from the
contract value. In comparison, the average preparation time of a Bank loan is
14.4 months.
6
Asian Development Bank ‘s Policies and Procedures on the Use of Consultants by ADB and
its Borrowers indicate 20-21 months as the duration of the Borrower procurement process as
against 7 months when the selection is conducted by ADB itself.
7
In 1995 the average prior review contract value was $ 1,214,000.
DURATION (MONTHS)
ACTIVITIES 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Advertising - 159 days
Submit RFP to Bank for No Objection 116 days
45 days
Submit RFP to Bank for No Objection - 23 days

Bank No Objection to RFP 8 days
14 days
Bank No Objection to RFP - 17 days

Issue RFP to Consultants 7 days
14 days
Issue RFP to Consultants - 47 days

Receive Technical & Financial Proposals 46 days
75 days
Receive Technical & Financial Proposals - 131 days

Submit TER to Bank for No Objection 73 days
14 days
Submit TER to Bank for No Objection - 38 days

Receive Bank No Objection to TER 10 days
14 days
Figure 2: Selection Process by Activities and Duration
Receive Bank No Objection to TER - 89 days

Contract Signature 54 days
75 days
31
World Bank estimated duration Average duration 97 assignments Median duration 97 assignments
32
Figure 2 shows that delays in the selection decrease as the selection process
advances: The longest delay is in the “short-listing stage” ending with Client’s
submission of the RFP (including the short list) to the Bank. This stage lasts
159 days in average (median 116 days). Even though web-based requests for
EoI have helped reduce submission time, the screening and review of on
average 21 EoIs per assignment remains very time-consuming and problematic.
The longest delays occur in cases where a Client with no familiarity with
selection procedures is required to evaluate qualifications of consultants with
whom he has very little or no previous experience. It should also be noted that
the average time allowed for proposal preparation, 45 days, is shorter than the
CSM estimate of 75 days.
The second phase of the selection, from receiving the proposals to submitting
the technical evaluation report to the Bank, takes 131 days on average (median
73 days). It is remarkable that the evaluation of proposals takes less time than
the preparation of the short list and the RFP. Increased familiarity with the
process and the focus on a limited number of submitted proposals perhaps
renders the process more manageable. The final part of the second phase, from
the Bank’s “No Objection” to the Technical Evaluation Report to the signature
of the contract, lasts 89 days in average. However, it should be observed that
the median of 54 days is below the 75 days estimated by the CSM.
The average duration of the complete selection process varies by region:
Table 2: Average Duration of Selection Process by Region

Region ECA EAP MENA SAR LAC AFR
Days 431 401 468 561 536 466
Months 14.4 13.4 15.6 18.7 17.9 15.5
The sample files suggest that the duration of the selection process is influenced
by several factors:
• Client’s low technical capacity and lack of familiarity with RFP
preparation;
• Large number of EoIs and volume of requested information;
• Insufficient time availability of EC members;
• Delays in the project’s implementation8;
• Absence of protocols regulating EC composition and evaluation
procedures;
• Inefficient approval processes at critical points, and incomplete or unclear
delegation of authority9;
8
Looking at the 15 cases with the longest delays from the sample study, only 3 assignments
pertained to implementation supervision of construction contracts.
33
• Political interference with the selection and approval process;
The mentioned delays often result in fragmented project implementation and

slow disbursement. Moreover, delays can have a negative impact on quality of
the services as they normally result in the fielding of experts of lesser quality
than originally proposed, because a consulting firm awarded the contract
cannot be expected to retain the same team for too long periods. Conversely, if
a consulting firm considers that delays are likely, it may be tempted, only to
ensure a high technical rating, to propose candidates with excellent credentials
whom it knows will probably not be available at the time of the project start.
Clients have first to deal with screening and evaluating in average 21 EoI of
varying dimension. They will then invite 5.3 proposals in average and receive
4.1 each containing 100-150 pages of technical text. Therefore evaluation
committees are faced with the review and evaluation of in total 400 to 700
pages of complex technical information. Thorough evaluating the submitted
material can easily take two to three weeks to each evaluator working on the
proposals on a 75% time basis.
It was not possible to arrive at a comprehensive estimate of the cost incurred by

firms in obtaining an award from the moment a decision is made to pursue a
Bank funded client’s assignment to the contract signature. International
consultants interviewed estimated that direct proposal preparation costs for
competed assignments under MDB procedures vary from a minimum of $ 25-
30,000 for simple proposals to a maximum of $ 60-80,000 for complex ones.
As the chances of success are about one in five, the fee of a successfully
tendered assignment has to sustain the cost of the other four presented
proposals. This may help explain why certain consultants abstain and why
others may try to resort to unfair and unethical shortcuts in order to secure the
contract.
8 Participation
For the 97 competed assignments of the study sample, an average of 21
expressions of interest was received. In total, 920 consulting organizations
were short-listed, alone or in association. The following figures show that the
average shortlist of the 97 competed assignments in the study sample includes
5.3 firms (Figure 3), the average number of received proposals is 4.1 (Figure
4), and 3 of them (57%), achieve the opening of their financial envelopes.
9
In countries facing donor or domestic pressures to improve governance (and reduce
corruption), the government approval process can become even more detailed and time
consuming.
34
Figure 3: Size of ShortList
Cumulative Frequency Distribution of Size of the Shortlist
100%
Cumulative Frequency
80%
97 Assignments
60%
Mean
40%
Median
20%
0%
0 1 2 3 4 5 6
Size of Shortlist
In slightly more than 80% of all cases, the short list consists of 4 firms or more.
The size of the short list is either 5 or 6 in almost 60% of the cases.
Figure 4: Number of Submitted Proposals
Cumulative Frequency Distribution of Number of Proposals
100%
80%
60% 97 Assignments
Mean
40%
Median
20%
0%
0 1 2 3 4 5 6
Number of Proposals
In roughly 60% of cases, the number of submitted proposals is 4 or less, and in

less than 20% of cases it is 2 or less. The number of submitted proposals is
either 5 or 6 in less than 20% of the cases (Figure 4).
35
As to be expected, the proposal submission rate increases when the number of

shortlisted firms decreases (Table 3):
Table 3: Proposal Submission Rates by Short List Size

6 firms 5 firms 4 firms 3 firms
4.4 (74%) 4 (78%) 3.5 (89%) 3 (100%)
The percentage submission rates can also be displayed by a cumulative

frequency distribution of submission rates (Figure 5). The results show that in
40% of the sample cases all invited firms submit proposals10 while in slightly
more than 20% of all cases only half of them submit proposals:
Figure 5: Percentage Submission Rates
Cumulative Frequency Distribution of Submission Rates
100%
80%
97 Assignments
60%
Mean
40%
Median
20%
0%
0% 20% 40% 60% 80% 100%
Submission Rate
The average proposal submission rate is 77% and the rate of admission to
opening of financial proposals is 57%. However, submission rates vary by
region as follows:
Table 4: Average Proposal Submission Rates by Region in the Study Sample

ECA EAP MENA SAR LAC AFR
70% 81% 79% 88% 62% 81%
A closer look at the submission rates by region reveals a conspicuously lower

submission rate in LAC (an eye opener!) than in the other regions. The reason
for these lower submission rates in LAC was not investigated and opinions
10
The figure displays these submission rates in the 90 to 100% spectrum. In reality they are all
100%.
36
among TTL and consultants vary. A closer look at the dropout cases in LAC
reveals that 51.5% of the dropouts were Part I consultants. 14 out of the total of
17 Part I consultants11 dropout cases pertained to assignments in Brazil and
Mexico where the local consulting industry is considered to be fairly capable,
competitive but also protected.
The Bank project portfolio embraces a much more widespread distribution of

professional and expert specializations (Table 1) than it did in the past,
nevertheless it may serve as an indicator of consultants’ interest in Bank
procurement that, of the 920 consulting firms shortlisted in the study sample,
only 9.6% belong to the 2004 list of the 50 world’s top consulting engineering
and architectural firms. If one expands the search to the 100 top firms the
percentage increases to 13.6%. Only 2.7% belong to the list of the world’s top
50 management and strategy consulting firms12. The same firms win a minimal
proportion of contracts when they do participate.
9 The First Phase of the Selection Process: Preparing the

Request for Proposals (RFP)
For the Client to select the most suitable consultant, the RFP must correctly
combine the Terms of Reference (ToR) with staff months estimate, evaluation
criteria, Minimum Technical Score, selection method, weight of quality, and
contract type. Each of these parameters requires calibration consistent with the
particular objectives and risk characteristics of the assignment’ ToR. If this is
not done correctly, the RFP can mislead consultants as of the information
required from their proposals, and the Client may form an imperfect picture of
a consultant’s suitability for the assignment.
The study sample shows that Clients generally specify and compose the RFP as
if its parts bore no relation to each other and not as an internally consistent
search mechanism. This is particularly evident in the description of the scope
of work and the related staff-months estimate or budget. Quite often the
specialist who has prepared ToR, staff moths or cost estimate has little role in
the specification of the RFP, the short list preparation and in the selection itself.
9.1 Terms of Reference
The ToR specify the nature and the objectives of the assignment, the type and
scope of the services needed, and the expected outcomes. The ToR are the first
11
Part I consultants are firms from non-borrowing Bank member countries.
12
The data are sourced from: “The Consulting and Architectural Groups, a Swedish and
International Survey, Swedish Federation of Consulting Engineers”, November 2005 and
“Vault Guide to the Top 50 Consulting Firms – Management and Strategy 2005 Edition” Vault
Career Library.
37
part of the RFP to be prepared, and the key one, as they identify the main
technical characteristics and risks of the consulting assignment, and influence
the specification of all the other selection parameters. For the purpose of this
study the ToR of all sample assignments were examined, classified and graded
according to those characteristics.
9.1.1 Grading of the ToR
The ToR of the study sample were first grouped under the headings:
(1) Project Preparation;
(2) Project Implementation; and
(3) Technical Assistance.
For each group, the ToR were then classified according to three categories of
risk, implied in the following considerations (para 3.2 of the Guidelines):
• Complexity/specialization, (Risk of design mistakes, cost overruns, delays)
• Project value and/or downstream impact, (Risk of economic losses, safety)
• Level of direct impact, (Temporal proximity of impact)
• Comparability of the proposed services, (Risk of poor selection)
Each ToR was then rated on a risk scale from 4 to 12, whereby each main
characteristic was given a score between one and three points. For example a
simple ToR with limited value and low direct impact was rated 4, while a
complex assignment combined with large direct downstream effects, or high
project value, likely to be proposed in ways difficult to compare, received a
maximum risk rating of 1213. This risk rating provides an approximate
indication of the potential consequences of mistakes of the consulting services
for the project. The grades were distributed as follows:
Grades 12 – 10 (High Risk): Very complex or complex ToR for high value
projects. ToR that are not detailed or sufficiently specific, making
proposals difficult to prepare. ToR in which the Client asks consultants for
new approaches, innovative solutions and methodologies likely to lead to
differing work plans. Technical proposals for these assignments are
difficult to compare and the downstream impacts can be direct, large, long
lasting and hard to mitigate. ToR where the related project value is affected
by the services’ quality. Common examples: Master plans, pre-and
feasibility, design studies of complex or/and specialized high value projects
(not only infrastructure); complex strategy studies, institutional reform
13
Some ToR may be borderline: in fact a ToR graded 9 and assigned QCBS as a selection
method could be easily considered a QBS case especially if the client does not want to trade
quality and price by means of a scoring formula.
38
plans and change of management advisory services, regulatory studies and

design for public services markets in all sectors.
Grades 9 – 6 (Medium Risk): Medium to low complexity TOR

requiring efficient and timely implementation of already known
approaches, solutions or methodologies. As work plans for these
assignments are not likely to differ too much, proposals should be fairly
comparable. Likely to belong to these grades are technical assistance
assignments of medium complexity in institutional and social fields,
implementation design or supervision of large but not complex
infrastructure projects requiring efficient execution of already specified
solutions (e.g. timely and accurate implementation and good management).
The potential negative impacts of these assignments are low and their
mitigation can be estimated.
Grades 5 – 4 (Low Risk): Well specified ToR for simple and low-impact
assignments requiring fairly common and explicit knowledge. Solutions
and methodologies are already known and sometimes even pre-packaged.
Proposed work plans should be comparable. This group includes
technology services, audits, inspection and surveillance services, awareness
campaigns, statistic surveys, simple field investigations; implementation of
simple technical assistance; humanitarian assistance services of a basic
nature. For these assignments, the value of the underlying project cannot be
markedly affected by the quality or the outcome of the services.
39
The following Screenshot 3 illustrates the method adopted for grading the ToR:
Screenshot 3: Grading the ToR
The rating exercise carried out for all assignments of the study sample suggests
that their respective ToR indeed show differentiated risk characteristics.
Therefore, in accordance with Bank Guidelines they require different selection
methods including a different weight for quality. Based on the above
definitions the sample assignments where graded as illustrated in Table 5.
Table 5: Assignments by Type and Risk Grade (Excluding SSS)

Risk Technical Project Implementation
Total
Grade Assistance Preparation Services
12 2 1 3
11 6 1 7
10 14 3 1 18
9 7 1 1 9
8 14 1 9 24
7 9 2 5 16
6 8 1 6 15
5 1 1 2
4 3 3
Total 64 11 22 97
40
One can question the decision of setting the borders of the risk grades between
9 and 10 and between 6 and 5. Obviously the decision is in part based on a well
considered judgment of the authors. Setting the borders somewhat higher (or
lower) would be justified by assuming more (or less) risk adverse Clients.
It is observed here that the present Guidelines, as well as past ones, do not
contain any reference to the need for risk analysis. Para 3.2 of the Guidelines
on “Terms of Reference” does not mention the risk or complexity level of an
assignment nor hint to a relation existing between ToR and the main process
parameters, such as the selection method, the type of contract and the short list
for preparing a balanced and effective RFP.
Most ToR examined, include a weak description of the assignment’s

deliverables, illustration of outcomes or explanation or measures of consultant
performance. They generally over-specify the inputs consultants must provide
and the compliance with the process minutiae. The same bias is applied to the
corresponding contracts “that are in essence input, “and “compliance based”.
The detailed specification of consultant inputs is tradition based, and originates
from supervision services for project construction where consultants fulfill
eminently agent’s tasks. With increasing complexity and specialization
required also in public sector projects, this approach typical of public contracts
increasingly limits the possibility for consultants of proposing innovative
solutions or alternative methodologies and work plans even if doing so could
benefit the quality of the services and the project.
The previous consideration points at the difficulty clients often have

understanding or thoroughly appreciating the details of what consultants offer
and provide them. The result of this asymmetry of information or knowledge
gap is major cause of concern as it may easily lead to wrong selection decisions
and also to corruption. The sample’s files correspondence often show that
Clients and Bank PS lack the necessary training to address the above
complexity with necessary flexibility and sound judgment for this type of
procurement.
9.2 Staff Months and Assignment Cost Estimates
The higher the risk grade, the more arduous is for the Client to indicate a staff
month estimate consistent with the ToR, and even more to prepare a reliable
cost estimate14. For the invited consultants a risky guessing game starts when,
14
In the 2004 Guidelines the Client can provide either the staff months estimate or the cost
estimate but, correctly, not both. Para 2.4 indicates that a “well-thought and thorough cost
estimate is essential if realistic budgetary resources are to be earmarked”: the main reason for
having a cost estimate in the RFP is another one, namely to guide the consultants on the desired
41
after reading the ToR, they try to figure out how the Client has arrived at these
estimates and how to respond with a fitting technical proposal.
The weakness of quoting in the data sheet a staff months or alternatively a cost
estimate without differentiating between national and international consultants
is exacerbated, as will be later discussed, when QCBS is applied to
heterogeneous shortlists because of the characteristics of this selection method.
Differing unit remuneration rates when using QCBS can transform the
selection into an inefficient and unfair price competition because the
composition of the estimate is not specified and/or the evaluators, as will be
seen in 11.1, give similar scores to different technical proposals. The attached
extreme, but real example of a QCBS case provides an illustration of this.
Table 6: Recent Example of Evaluation Outcome

Technical Score Price Financial Score Combined Score
(Firm A) 74.49 1,81M 100 79.59
(Firm B) 83.36 3,54M 51.21 76.93
(Firm C) 81.25 3,95M 45.89 74.18
(Firm D) 82.68 6,13M 29.54 72.06
(Firm E) 80.48 3,29M 55.01 75.38
In this example the RFP indicated 300 as the estimated key staff months
without any further distinction between international and domestic, and 70
points as the Minimum Technical Score. The question remains how the Client
composed this estimate and the short list, and how comparable the technical
proposals really were.
9.3 Evaluation Criteria and Minimum Technical Score
After ToR, staff months or the cost estimates have been established, the Client
decides the selection parameters, in particular:
• the evaluation criteria and their score ranges,
• the minimum technical score (MTS),
• the selection method,
• if QCBS is chosen, the relative weight of quality,
• the factors determining the short list composition and
• the type of contract.
The correct determination of these parameters allows the Client to ask

consultants precise questions, measure and weigh their answers in a balanced
cost of the contract. If the client must reveal a budget the ToR need to be precise and FBS
should be adopted.
42
manner allowing him to discover the proposal that best matches the proposed
ToR. A well specified and coherent RFP would allow consultants to understand
exactly what is required and encourage them to provide the information needed
(i.e. the proposal) to demonstrate their suitability for the assignment.
9.3.1 Setting the Evaluation Criteria
Para. 2.15 of the Guidelines and the Standard RFP Data Sheet recommend the
following standard evaluation criteria and scoring ranges:
Table 7: Standard evaluation criteria and scoring ranges

1. Consultant’s specific experience 0 to 10 points
2. Methodology (Approach, Methodology & Work Plan) 20 to 50 points
3. Key Personnel (General Qualifications, Adequacy, Experience 30 to 60 points
in Region & Language)
4. Transfer of Knowledge 0 to 10 points
5. Participation by Nationals 0 to 10 points
6. Total 100 points
The study shows that Clients stay within the Guidelines recommended score
ranges in 90% of the cases but do not consider the risk characteristics of the
ToR when distributing maximum and minimum scores. A high risk ToR should
require a stronger emphasis on methodology and work plan, as recommended
in para. 2.16 of the Guidelines. In all reviewed cases of high risk or complex
assignments (ToR graded 10 to 12) the average score awarded to the
methodology is only 33.5, whereas the Guidelines allow up to 50 points.
Clients allocate considerably more points to “key professional staff” than to

“approach, methodology and work plan”. In 65% of all reviewed cases, 50 or
more points are assigned to key staff while in 53% of cases only 30 or less
points were awarded to approach and methodology.
Relaying too much on proposed on key professional staff can be debated, as the
delays mentioned in Chapter 7 can put the robustness of this criterion in
question even if consultants are left with the burden of proving equivalence in
case of substitutions (sometimes unfairly).
The Guidelines’ approach of placing importance on the qualifications of

proposed key staff, not as a team, but as a sum of individual CV scores is also
questionable. As a consequence, a “customized” set of individual CVs can win
over firms operating mostly with well tested teams, even if these individuals
may perform poorly for lack of experience in working together. Finally,
43
another negative aspect of this restrictive focusing on individual CVs for

narrowly specified tasks or roles, and price, is that ToR don’t encourage or
request the participation of senior professional staff from the consultants’ home
office.15 Ignoring these two factors can sap consultants’ interest in the
assignment and probably also leads to a loss of value for the Client when the
assignment does not progress as expected.
It is also noticeable the evaluation criteria focus on inputs without mentioning

the expected results or the outcomes of the services. A change of evaluation
criteria and scoring ranges listed in Table 7 could help rectify this situation.
9.3.2 The Minimum Technical Score
The Guidelines do not provide suggestions for setting the Minimum Technical
Score (MTS) except for the case of a LCS.16 Based on the study sample the
MTS is set as follows:
Table 8: Frequency distribution of Stmin in QCBS

Minimum Technical Score (Stmin)
MTS 60 65 70 75 80
% 10% 2% 40% 44% 4%
Minimum Technical Score - Summary Statistics
Mean Median Standard Deviation
73.43 70.00 4.8
In 84% of the reviewed cases Clients opt for a MTS of either 70 or 75 points. It
would also be reasonable to expect that Clients correlate the MTS with other
parameters such the risk/complexity grade of assignment ToR, including the
impact of the services on the underlying project. In reality, the correlation
coefficient between the grade of the ToR and MTS in the sample is positive but
only slightly above zero. The average number of technical proposals scoring
above the MTS is 3 out of 4.1 proposals presented (based on 97 assignments),
or 57% of all consultants short-listed.
15
As the sampled assignments are competed on the basis of QCBS consultants avoid including
such staff in the proposals unless the Client explicitly specify them, which almost never
happens.
16
Para. 3.6 referring to Least – Costs Selection (see footnote 31 of the Guidelines) indicates
that for LCS the Minimum Qualifying Mark shall be 70 points or higher.
44
9.4 Choice of the Selection Method
According to the Bank’s procurement statistics the following selection methods

were adopted for the 1972 “prior review” contracts awarded in FY 03/04:
Table 9: Selection Methods for Prior Review Contracts in FY 03/ 04

QCBS SSS FBS CQS QBS LCS
66.46% 20.11% 3.83% 4.03% 4.54% 1.12%
Selection Methods for Assignments where Competition Took Place
QCBS FBS CQS QBS LCS
90.82% 4.08% 2.04% 2.04% 0.88%
The study sample exhibits a similar frequency distribution:
Table 10: Selection Methods for Sample Assignments (114)

QCBS SSS FBS CQS QBS LCS
88 17 4 2 2 1
77.20% 14.90% 3.50% 1.75% 1.75% 0.88%
Selection Methods where Competition Took Place (97 Assignments)
QCBS FBS CQS QBS LCS
88 4 2 2 1
90.72% 4.12% 2.06% 2.06% 1.03%
QCBS is used for almost 91% of assignments where competition takes place.
This indicates that perhaps lacking adequate training, Clients opt for QCBS
without regard for risk/complexity of the ToR, project value, and comparability
of proposals and composition of the short list.
Considering that para. 1.5 of the Guidelines recommends that “the selection is
based on the quality of the proposals and where appropriate, on the cost of the
services”, the reasons for preferring QCBS requires consideration. Clients
appear to recur to QCBS compulsively for lack of sufficient experience in
evaluating quality and to avoid the suspicion of discretion. In many Part II
countries, selection of consultants based on “least cost” and contract
negotiations with “bargaining” are found to be quite frequent. The perception
of low paid Client’s staff that consultants are overpaid, coupled with a lack of
appreciation for the value of independent of the advice provide additional
reasons for “squeezing the consultants”.
The 1997 Guidelines and their subsequent editions use QCBS to describe the
selection process and the 2004 edition call QCBS “the most commonly
recommended method” (para 1.5 the Guidelines, sentence 3). This may
reinforce the Clients’ latent preference for QCBS. The interviews with Bank
staff confirm that the wording of the Guidelines and the advice, or lack thereof,
45
of procurement staff contribute to leading Clients in this direction. In contrast,

QBS is used in less than 5% (versus 32% of high risk assignments found in the
sample) of all contracts awarded during FY03/04, while FBS and CQS (after
their introduction in 1997) appear to meet with little acceptance. Clients are
very reluctant to use LCS even when the characteristics of the assignment
suggest it.
The 20% quota of SSS includes awards to NGOs, universities, research

institutes and other organizations providing various services to Bank Clients.17
The percentage of SSS awarded to consulting firms represents only 3.4% of the
total. According to TTL and PS interviewed, compulsive tendering can be the
norm also when simple cost/benefit analysis shows that it is not adding value to
the project, sufficient to justify the extra cost but to fend off suspicions of
collusive practices. This happens also when repeat orders, continuations or
unique assignments could be awarded more efficiently on a sole source basis
and in accordance with para. 3.9 of the Guidelines.
The interviews with Bank staff confirm a widespread belief that prices of
services can be directly compared independent of the assignment
characteristics, comparability of proposals and value of the underlying project.
It is therefore not surprising that Clients’ preferences for QCBS are almost
never objected.18
9.4.1 Guidelines Para. 3.2: Appropriate Selection Method for Complex

Assignments
The sample statistics show that the Clients adopted QCBS in almost 91% of the
competed cases, although para. 3.2 of the Guidelines recommends QBS for
complex or highly specialized assignments with high downstream impacts that
can be carried out in substantially different ways. Based on the grading of the
ToR, (see 9.1.1), 32% of the observed assignments earn a grade from 10 to 12
and satisfy the Guidelines’ conditions for QBS. This percentage increases to
41% if assignments graded 9 are included among those labeled “risky and or
complex”. In fact, only in two of the 97 competed cases Clients adopted QBS.
It is troubling to observe that in all these cases the Bank “No Objection” was
provided without any question or warning to the client.
17
In FY 1978, 50% of Bank funded assignments were awarded on a SSS basis.
18
Procurement Plans annexed to the Project Appraisal Document show in general the list of
Consultant contracts to be awarded in the course of the project. Although at that point the ToR
for the contracts’ assignment is not yet available QCBS is the selection method generally
indicated or suggested. Many TTL have declared that this part of the PAD is left to PS staff
without any discussion on the choice of the selection method taking place.
46

Assignments with Mixed Short Lists
Para. 2.8 of the Guidelines prescribes to ensure fair competition, that “the
shortlist should preferably comprise consultants of the same category, similar
capacity, and business objectives”. Should a short list mix other organizations
such as NGOs, government agencies, research institutes and foundations with
typical consulting firms, para. 2.8 recommends that QBS or CQS (for small
assignments) be applied instead of QCBS.
In the study, in 35% of competed cases Clients’ short lists mix consultants with
“other organizations”, but disregard the requirement of para. 2.8, and the
possible consequences for the fairness of the competitive process. Also, the
ensuing price dispersion is not considered a factor of concern.
9.5 Setting the Weight of Quality
According to para 2.1 of the 2002 Guidelines, “the relative weight to be given
to quality shall be determined for each case depending on the nature of the
assignment”. Para. 2.22 further states that “the weight for ‘cost’ shall be
decided by taking into account the “complexity of the assignment and the
relative importance of quality”. The same paragraph provides that for the
assignments requiring QCBS “the weight for ‘costs’ shall ‘normally’ be in the
range of 10 to 20 but in no case shall exceed 30 points out of a total score of
100.”
In the sample cases in which QCBS was adopted, the relative weight of quality
α, in average 0.78, is distributed as follows:
Table 11: Frequency Distribution of α in QCBS
Weight of Technical Proposal (α)

α = 0.70 α = 0.75 α = 0.80 α = 0.85
10% 16.5% 66% 6.5%
The sample refers to FY 03/04 when the 2002 Guidelines where in place.
The 2004 Guidelines tightened the instruction indicating that “the weight for
cost should normally be 20 out of a total score of 100”. This has probably
further narrowed the distribution and placed the mean α closer to 0.80 .
In the sample Clients tend to define the weight of quality without considering
the “complexity of the assignment or the relative importance of quality”
suggested by para. 2.23. Contrary to logic, in 30% of the reviewed complex or
high risk assignments (where the ToR have been graded between 10 and 12)
47
weights of 25 or 30 were allocated to price. In the remaining 70%, 20 points

were assigned to price.
9.6 Choice of the Contract Model
Para.4.1 of the Guidelines directs users to adopt the Lump-Sum contract model
when the assignment is simple and the output of the services and their related
payments can be tied to each other19. When the assignments are risky or
complex para 4.2 advises to use the Time-Based contract20.
Notwithstanding the guidance, one observes that in the sample, as in the case of
the other already discussed parameters, Clients offer either forms of contract
independently from the risk or complexity characteristics.
Table 12: Sample Assignments by Contract Type

Risk Grade of ToR Time-Based Lump Sum
10-12 12 15
7-9 30 20
4-6 8 12
Total (97 cases) 50 47
Total (%) 52% 48%
This indifference causes that consultants are proposed Lump Sum Contracts for
risky assignments that expose them to the possibility of delayed or even lost
payments, for reasons over which they may have little, or no control. Contrary
to what one would expect, quite a few simple, low risk assignments receive
Time-Based Contracts. A common characteristic of both contracts, and perhaps
a cause of confusion between them, is that both focus strongly on compliance
and obedient behavior rather than quality of services and deliverables as
measures of Client’s satisfaction.
9.7 Consideration for Competition: The Short List
The third consideration of the Guidelines (para 1.4) requires borrowers “to give
all qualified consultants an opportunity to compete”. They further indicate that
the Client’s RFP for the assignment shall be sent a short list of consultants with
(para 2.6) a “wide geographic spread” (para.2.6) and include “not more than
two consultants originating from the same country, while at least one should
originate from a developing country (if a qualified one is available)”. It appears
19
Lump Sum Contracts should be used for simple planning and feasibility studies, detailed
design of standard or common structures, preparation of data processing systems etc.
20
Time-Based Contracts should be adopted for complex studies, supervision of construction,
advisory services, and most training assignments.
48
that the Guidelines are more concerned with the geographic origin of the
shortlist that with its quality.
With respect to fairness, it is also required (para 2.8) that the short list be as
much as possible composed of consultants of “same category, similar capacity
and business objectives” and that, lacking this possibility, consultants should
be selected on QBS or CQS. These rather vague requirements are not easily
understood and fulfilled. They often also contradict the requirement of a “wide
geographic spread etc.”, because of the different organization and cost structure
of consulting firms in Part 1 and Part II countries.
Insufficient knowledge of the market, inexperience in evaluating consultant’s

qualifications and agency problems, create opportunities for corruption at the
point of short-listing, involving the consultants and the bureaucracy, or the
government itself.
The 159 days needed by Clients preparing a suitable short list and the RFP,
together with the low participation rates of invited consultants suggest that
clients have serious difficulties interpreting and satisfying Bank requirements
on short listing that need to be addressed.
9.7.1 The Request for Expressions of Interest (EoI)
Starting with the 1997 Guidelines, Clients must first publish a Request for EoI
in a “Specific Procurement Notice” (SPN). The publication of the request
provides fairness and transparency, at least formally, to the first part of the
selection process, and allows Clients to collect information from consultants
that they may not know or that may not be aware of this opportunity.
For the competed sample assignments, in average 21 consultant organizations

responded to Bank Clients request for EoI.
The Client lacks expert staff to conduct professional reviews of unknown

consulting firms, complicated by language, geographic and cultural problems.
Conversely, it is easy and inexpensive for consultants to submit a large and
complex EoI from their existing electronic files, thus raising the question of the
“sincerity” of the EoI itself. A similar problem can arise for consultants,
namely the one of Client’s sincerity, as this may already have in mind a list of
consultants even before issuing the request of EoI especially if the assignment
is mission critical.
In about 40% of the sample cases, the document files examined contain
extensive short-listing reports, although they, strictly speaking, are not required
by the Guidelines. Clients are led to prepare these large documents confused by
49
the fact that certain donors (including the EU) require the submission of formal
“prequalification reports”. These reports focus on technical parameters that
allow consultants’ technical qualifications to be quantified and added up, but
cannot capture factors such as integrity record and trustworthiness of
candidates because of past work for this client. The need of matching the short
list with the ToR and other RFP parameters is not considered, and it is obvious
that some Clients prepare these reports only to satisfy transparency and
compliance concerns.
Their value, or the extent to which the reports help overcome information
asymmetries between consultants and clients and help prepare a well fitted
shortlist well matched to the client and its ToR, does not compare with time
and effort spent for preparation.
9.7.2 Homogeneous Short List with a Wide Geographic Spread
The study shows that Clients include in their short lists in average 5.3 firms and
that the geographic spread objective is “somehow” satisfied in the majority of
the cases requiring international competition, as shown in Table 1321.
50
Table 13: Short-listed Consultants from Part I and Part II by Country (Study
Sample)
Part I Part II Borrower Part II Others
Country No Country No Country No
UK 106 Indonesia 45 India 12
USA 86 Pakistán 43 Bangladesh 12
Germany 57 Brazil 41 South Africa 9
Canada 40 India 30 Turkey 4
France 37 Turkey 21 Egypt 4
Netherlands 31 Bangladesh 16 Thailand 3
Australia 31 Tanzania 15 Costa Rica 3
Spain 19 Russia 14 Philippines 2
Denmark 19 Uganda 13 Jordan 2
Sweden 18 Lebanon 11 Zimbabwe 2
Italy 11 Nigeria 9 Pakistan 2
Norway 8 Zambia 8 Ukraine 1
Switzerland 8 Philippines 7 Colombia 1
Belgium 7 S.Africa 6 Panama 1
Finland 6 Bolivia 6 Lesotho 1
Japan 6 Serbia 5 Albania 1
Israel 5 Croatia 5 Slovenia 1
Ireland 4 Argentina 5 Peru 1
Hong Kong 3 Afghanistan 5 Iran 1
Austria 2 Romania 5 Sri Lanka 1
New Zealand 2 Ghana 4 Macedonia 1
Greece 1 Vietnam 4 Ghana 1
Others - Others 25 Others 4
55% 507 37 % 343 8% 70
Clients often struggle preparing a homogeneous or fair short list of qualified

candidates in combination with the required geographic spread, especially
when they are satisfied with those that they already have identified over time,
or are already working with them in a mutually satisfactory relationship on or
related issues..
Insisting on wide geographic spreads can lead to Clients inviting less qualified
consultants that they do not know well, and that have little chance of being
awarded the contract “in a professional way”. This forcing of geographic
considerations can compromise the integrity of the selection process and
eventually its outcome, especially in the case of inexperienced Clients facing
unscrupulous consultants, and vice versa.22
22
In several instances proposal Evaluation Reports indicate under the heading “consultants
specific experience” that the consultant does not appear to have “sufficient experience” to
51
9.7.3 The case of “Other Organizations”
Para 1.3 of the Guidelines (starting with the 1997 edition) includes under the
term “consultants” a wide variety of public and private entities that
occasionally provide consulting services. Organizations “other than
consultants” are NGOs, universities, government agencies23 and public
institutes, all of which have both different business objectives and cost
structures than private consulting firms. According to the study 35% of the
organizations appearing in the sample shortlists can be classified as “other than
consultants”
According to the study findings these “other organizations” were distributed as

follows:
• Non-profit research institutes 25.0%
• Non-profit NGOs 25.0%
• Government agencies 7.5%
• Universities 15.0%
• Commercial banks 5.0%
• Foundations 7.5%
• Advertising agencies 2.5%
• Others 12.5%
Total 100.0%
The high level of “other organizations” participation contributes to the
complaints of unfairness and frustration voiced by private consulting firms also
claiming that in cases of mixed short list the selection method should not
include price as an upfront factor of selection (QCBS, LCS) as indicated in
para 2.8 of the Guidelines.
10 The Evaluation Committee

For the Client, the probability of selecting the most suitable consultant depends
on one hand on the balanced matching of the ToR with RFP and short list, and
on the other, on the technical capacity and ethical conduct of the EC.
For consultants, even when the quality of the RFP is high, trusting the EC
capacity of evaluating professionally and impartially remains the fundamental
concern. What happens with their labor intensive and expensive proposals after
submission is a justified preoccupation. The review of the evaluation reports
discussed in Chapter 11 confirms that the relationship between the quality of a
technical proposal and the score assigned to it by members of the EC can be
undertake the proposed assignment; this implies that the Client did not know the capabilities of
the firm when the shortlist was prepared.
23
In further 12% of all reviewed assignments, the shortlist included only NGOs.
52
quite weak. As the technical complexity and specialization of the services

required increase, the information asymmetries between Client and consultants
also increase. The competence of the EC to adjudicate proposals is being
stretched along with the credibility of the process to be effective and fair.
The Guidelines refer simply to “the Borrower” without articulation of the

Client organization. Nevertheless the correspondence of the sample cases
testifies to intense transparency, trust and compliance control issues that can
arise between the EC (the bureaucracy) and the government (or the politicians)
during the selection process, as the first retains information from the second
and the second tries to exercise its control on the first. These tensions can affect
the credibility of the selection process.
The level of experience of individual EC members and the manner in which

they evaluate, interact and make decisions is normally not disclosed in the
evaluation reports to the Bank. The proposal evaluation reports of the study
sample identify most members of each EC by name, rank, or position in their
organization, but do not provide criteria or reasons for appointing them. In very
few cases they specify detailed procedures or norms (e.g. ratings of criteria)
followed by in the evaluation or indicate if EC members were provided with
guidelines on how to evaluate and score quality. In most cases the EC is
appointed shortly or just before the evaluation begins and often EC members
had no role preparing the RFP including the shortlist.
ECs include executing agency officers ranking from directive (mainly

department heads and higher) and managerial to medium technical levels, with
titles suggesting backgrounds in areas more or less directly related to the
assignment.
Table 14: Presence in Evaluation Committees of the Study Sample

EA Department Director 27%
EA/PIU Project Manager 12%
EA Sector Specialist 38%
Supervisory or Line Ministry 35%
Procurement Specialist 20%
Independent Member 7%
N.A. 37%
Sector specialists presence in ECs is smaller (38%) than one would expect
considering that the proposals are generally large and complex. In 20% of the
sample cases the EC includes a procurement specialist, not necessarily expert
in consultant selection. In only 7% of cases, the EC includes an independent
member (appointed by third parties e.g. donors). Although these individuals
can add an independent vote in the EC, in order to be truly beneficial they
53
should also be truly experienced in selection of consultants and in the field of

the assignment. It should also concern the Bank that in 37% of cases, the
evaluation report does not identify the evaluators at all.
Supervisory authorities and directors are involved in 62% of the cases (often
chairing the EC), although their knowledge of the specific consulting
assignment may be superficial. Also the attention they can dedicate to the
complexities of technical proposals is limited. The reason for their presence in
the EC as evaluating members is normally to exercise control. However, their
presence often intimidates lower ranking evaluators.
Examining the sample evaluation reports, one observes different patterns of

scoring across evaluators. Some reports show fairly uniform technical scores,
while others display considerable disagreement when evaluating the same
proposal. “Decisive influence” concerning the rank order of technical scores
occurs in a fairly large number of cases (16%), as summarized in Table 23 in
11.2.2. This suggests that the manner in which the ECs is appointed, set to
function and supervised, needs to be internally more transparent and
accountable.
Finally, the sources of external pressure on the EC decisions vary from higher
authorities disagreeing with ECs decisions and delaying their clearance, to
competing consultants (or their representatives) trying to influence the EC or
the higher authority. The study files and testimonies suggest that these
pressures are indeed frequent. The average of 131 days (median 73) elapsing
between submission of proposals and the transmission of the evaluation report
to the Bank, are an indicator of the difficulties that ECs member meet
delivering decisions, as well of the “hesitation” of the higher authorities in
endorsing them.
A preliminary conclusion, at this stage of the study, is that EC capacity level,

reliability and its inner functioning appear as the main “black box” of the
selection process. The discussion of the manner in which technical proposals
are evaluated, discussed in the next chapter 11, suggests that these concerns for
the majority of the sample cases, are valid.
54
11 The Second Phase of the Selection Process: Proposal

Evaluation
Not everything that can be counted
counts, not everything that counts can be
counted.
Albert Einstein
11.1 The Evaluation of Technical Proposals
Consistent with the Bank statistics of prior review awards, the study sample
shows that 91% of all assignments are competed on the basis of QCBS. In 11%
of the QCBS cases, only one financial proposal was opened. For the remaining
cases, the following joint distribution of score ranks was observed:
Table 15: Sample Award Distribution

Financial Technical Score Rank Total
Score Rank Abs 1 % Abs 2 % Abs 3 % Abs 4 % Abs %
1 27 35 16 21 7 9 2 2 52 67
2 15 19 1 1 1 1 - - 17 22
3 4 5 - - 1 1 - - 5 6
4 4 5 - - - - - - 4 5
Total 50 64 17 22 9 12 2 2 78 100
The highest technical score is awarded the contract in 64% of cases. In the
remaining 36%, the award goes to lower ranked technical proposals (second,
third and forth ranked)24. Conversely, the highest financial score i.e. the lowest
price is awarded the contract in 67% of cases, with the lowest price being also
the highest technical proposal in 35% of them. If one assumes that quality and
price of winning proposals should be positively correlated, the above results
can represent quite a disturbing surprise. Furthermore, in the study sample the
correlation coefficient between technical and financial proposals scores is
negligibly small (ρ =0.09) although positive.
24
The ADB has reported in 2005, that the switching in rank due to the introduction of QCBS
changes the contract award in 20% of cases in their evaluations. It may be that , all other things
remaining the same, ADB staff does prepare more homogeneous short lists than Bank clients.
55
11.1.1 Analyzing the Award Distribution
Looking into the individual cases of the QCBS sample to ascertain reasons and
dynamic for the success rate of low priced proposals, it is useful to look at the
characteristics of the winners.
Table 16: Awards % Distribution by Origin & Technical Score Rank

Technical Score Rank 1st 2nd 3rd 4th Total
National Firms alone 23 4 4 1 32
Nat Firms Leading with Intl. Partcp. up to 50% 2 - - - 2
Intl. Firms with 100 to 50% Local Participation 15 12 5 1 33
Intl. Firms with 1 to 49% Local Participation 8 2 - - 10
Intl. Firms alone 16 4 3 - 23
Total 64 22 12 2 100
National firms participating alone win 32% of the awards, and make very little
use (2%) of international support. On the contrary international firms make
very large use of local professionals either employing them directly or through
local associates. As the QCBS rule does not require the RFP to specify the
Client’s suggested distribution between international and national staff months,
international firms crowd-in local consultants to lower their financial proposals
(in 4% of cases the international firms provide only the name while the entire
key staff is local).
In the sample the mean dispersion of technical scores is 1.1225 while the mean
dispersion of prices is 1.79. This suggests that ECs give similar scores to
technical proposals perhaps because they have difficulties observing quality
from proposals and scoring it, or because they don’t care much about quality
itself. The same proposals show very different team compositions of PartI/Part
II consultants which are quite differently priced. As a combined result, the
scoring rule often awards the contract to the lower priced proposals as shown in
Table 15.26 In fact in the study sample the median remuneration rate of Part I
consultants is $16,700 per staff month, compared to $3,100 of consultants from
Part II countries.
The mean dispersion of quality scores does not change significantly if one
separates the complex assignments from the simple ones (Table 17). Instead,
the price dispersion gets even higher for proposals of complex assignment.
25
The average technical score dispersion (Stmax/Stmin) for all evaluated proposals including
those that where rated below the MTS is 1.30.
26
Evaluators observing the staff month distribution of proposals can form an idea of their price
ranking and based on this influence the technical score of the proposals.
56
Table 17: Risk/Complexity versus Technical Score and Price Dispersions

Dispersion Coefficients Technical Score Price
Risk/Complexity Grade (12-10) 1.13 2.10
(9-7) 1.12 1.67
(6-4) 1.13 1.60
Average 1.12 1.79
Evaluators do not observe increasing differences in the quality of technical

proposals with increased risk/complexity, although the price dispersion
increases with grade. As a result price plays a more important role in the
selection when the complexity of the assignment increases. In fact the award
goes to the lowest bidder in 80% of the high risk/complexity cases. This should
be a major cause of concern as it shows how price increasingly influences the
final ranking as complexity of the ToR increases. Para 3.2 of the Guidelines
prescribes the use of QBS when assignments are complex just to avoid this
effect. Too much attention to price competition and too little to quality is
probably one of the main reasons discouraging many consultants from
participation (chapter 8).
11.1.2 High-and Low-Ballers
In the sample, in 5% of cases the highest ranked technical proposal with also
the highest price receives the award. Of them, only one case (nr.3) stands out
for receiving a very high score compared to others, and the highest price.
Looking at the cases individually, some appear consistent with a shill bidding
pattern, although it typically does not succeed.
Also to be mentioned is that only in 2% of cases consultants with the highest

financial score and the 4th technical rank get the award, and in 9% the highest
financial but 3rd ranking in quality receives the award. These cases were
individually examined. They can be characterized as successful “low balls” that
get barely over the MTS and show a very low price. Obviously there may be
more “low balls” that do not make the MTS.
57
11.1.3 Using QCBS in lieu of QBS
Using QCBS in lieu of QBS implies for the sample an average loss of 3.79
quality points:
Table 18: Frequency and Extent of Technical Score Loss ( dSt ) due to using
QCBS in Lieu of QBS
small intermediate high All
dSt ∈ (0,2] dSt ∈ (2,5] dSt > 5 dSt > 0
# of Cases 9 11 9 29
Frequency 31% 38% 31% 100%
MeanLoss 1.47 3.093 7.14 3.79
Whether this point loss is significant depends entirely on the characteristics of

the assignment and of the underlying project, as described in Chapter 9. If
QCBS is used across the board as done at present, the Client may be incurring
risks not worth taking, particularly in projects that have a high degree of risk
and complexity, or high value. As it were, of the sample cases in which the
price dominates the technical ranking, 32% turns out to be complex (or risky)
assignments for which according to the grading exercise carried out on the ToR
(9.1.), QBS should have been applied if the Guidelines had been adopted
correctly.
11.2 Analyzing the Score Distribution of the Sample
The purpose of the analysis of scoring patterns is to detect any bias of EC

members in the evaluation of technical proposals and to explore how the
outcome of a selection would have changed if some of the changes proposed in
the study is implemented27. Parts of the score sheet that records and analyzes
the individual scores given by five evaluators (Annex 2) are reproduced in
Table 19 for one of sample cases28. The individual scoring patterns reveal the
possible biases. Typically, a biased evaluator will give higher scores to his
favored consultant to make him look better or unduly low scores to the rival
consultants to make them look worse.
27
The data files of the study contain one Excel worksheet for each assignment (see Annex I).
This worksheet contains all available information, including a full record of each evaluator’s
technical scores. For each assignment it also includes an analysis of each evaluator’s scoring
patterns
58
Table 19: Mean and Standard Deviation of Scores by Individual Evaluators

Summary
Evaluation Criteria Evaluator Statistics
A B C D E Mean Std.Dev.
Specific Experience 8 8 8 7 7 7.6 0.55
Methodology/Workplan 17 15 15.5 16 16 15.9 0.74
Key Professional Staff 47 48 50 45 47 47.4 1.82
Transfer of Knowledge 8 8 8 8 7 7.8 0.45
Sum 80 79 81.5 76 77 78.7 2.22
In analyzing the study sample distribution of scores it is useful to know the

average rank order of technical scores, from highest (rank 1) to lowest (rank 6),
which is summarized in Table 20. The standard deviation is clearly increasing
as one moves from high to the lower score ranks. This indicates that there is
more dispersion in the scores of the lower ranking proposals.
Table 20: Mean Tech. Scores of Different Score Ranks (highest: 1 to lowest: 6)
Summary Technical Score Rank
Statistics 1 2 3 4 5 6
MeanScore 86.50 80.90 76.74 71.95 66.51 61.39
Stdev 6.23 7.92 8.61 12.15 13.09 14.09
CorrCoeff –0.98
This suggests a negative correlation between the score rank and the measure of
dispersion of scores. Table 20 displays the mean scores of different score
ranks, from rank 1 (highest score) to rank 6 (lowest score), and the standard
deviation of the associated technical scores. Clearly, the top score ranks (say
rank 1 and rank 2) are less dispersed than the lower ones (say rank 5 and 6),
and the correlation coefficient is –0.98. This suggests that evaluation
committees tend to do a better job at sorting out differences concerning those
technical proposals that are considered more valuable and rank higher.
11.2.1 Detecting Disagreement
Individual evaluators score differently for several reasons:

• Some evaluators may be either more generous or more stingy (generosity
bias);
• Some evaluators are inexperienced in consultants selection or in the field of
the assignment, although no ill intentions are involved, and stick to their
59
different opinions (persistent disagreement);

• Some evaluators may be biased in favor of a particular consultant due to
corruption, or because they favor a consultant they particularly esteem, and
use their scores to exercise influence (corruption bias).
One useful tool to assess such differences was to look at the dispersion of
technical scores across evaluators, which is measured in Table 20 by the
standard deviation of technical scores. A high standard deviation indicates a
high level of disagreement. A method was developed in this study to visualize
the scoring pattern of individual evaluators and is described in Annex 3.
Comparing the display of the original and the purified scores in Annex 3, one
sees that the generosity bias obscures the presence and extent of disagreement
between evaluators. Once that bias is cleaned, true disagreement between
evaluators over the ranking of proposals becomes clear and evaluators if
willing can try to deal with it. This purification is meant to render the
preferences of each individual more visible, which facilitates the comparison
and reconciliation of scores among members.
11.2.2 Detecting Decisive Bias
If an evaluator has been bribed to exercise influence in favor of a particular

proposal, that evaluator will try to inflate the score of the favored one relative
to the scores of other proposals. Of course, the favored proposal may win
independent of how the bribed evaluator gives scores. In that case, corruption
does not make a difference.
In order to detect possible corruption that makes a difference, one should check
for decisive influence of evaluators and groups of evaluators. An evaluator has
exerted decisive influence in favor of a particular proposal if that proposal wins
only due to the technical scores given by this evaluator. In that case, one would
check whether a particular evaluator has been decisive in selecting a proposal
as the one with the highest technical score.
As a routine procedure, the sample data sheets contain a check for decisive
influence concerning the highest technical score. This is done by computing the
rank order of technical scores that would have occurred if the scores of an
individual evaluator had been neglected in the computation of the average
score.
The study finds that decisive influence concerning the rank order of technical
scores occurs in a fairly large number of cases, as summarized in Table 21.
60
Table 21: Share of Evaluators with Decisive Influence in the Study Sample
Scores Mean Stdev
Unpurified 9.4% 0.49
Purified 15.89 % 0.44
This suggests that one should change the rules for computing the technical
score in such a way that decisive influence is neutralized. It is shown how to do
that in Section 11.2.3 as well as in Annexes 2 and 3.
If corruption is widespread, it is more likely that several evaluators are being

bribed, and corrupt influence is exerted by the combined scores of the corrupt
evaluators. In order to detect such concerted corrupt influence, one has to
check for coalitions of evaluators who exert decisive influence. The extension
of the above procedure to do this is straightforward; it was developed in the
course of the study and is shown in Annex 3. In the extreme it may occur that
all evaluators have been bribed. Such complete corruption cannot be detected
by looking at the score distribution. However, one has to keep in mind that
corruption becomes more risky and more costly when several evaluators are
bribed. Therefore, checking for decisive influence of one or two evaluators
should pick up or prevent a significant portion of corruption cases.
11.2.3 Tools for neutralizing outliers
A simple way of reducing the effect of outliers i.e. evaluators that score far
way from the rest, is usual to compute the technical score as a simple average
of the scores given by individual evaluators. “Simple” means that each
evaluator’s score is given the same weight. Yet, in order to neutralize decisive
bias, the client could when specifying the RFP, give less or even no weight to
outliers.
For instance, a simple implementation of this idea is to remove, for each

proposal, the highest and the lowest quality score in each score criterion, and
then compute the mean of the remaining scores (Annex 3). Of course, this
should only be done if the EC has at least 4 members.
Going one step further, one can compute the median of the quality scores of all
evaluators. The median is the quality score of the evaluator who is “in the
middle”: one half of the evaluators rate the proposal higher, the other half rate
it lower than the median evaluator. Unlike the mean, the median is not affected
by outliers. An extreme evaluation by a single, biased evaluator (possible
because he has been bribed) has no effect on the median score. A Monte Carlo
simulation of median aggregation was conducted and is illustrated in Annex 4.
61
11.2.4 The Importance of EC’ Collaboration
As shown in Section 11.2.2, a significant percentage of the individual

evaluations in the study sample (15.89%) is affected by biases that remain
unresolved, but have a decisive effect on the evaluation result. Lack of EC
accountability exposes the selection process to the risk that members,
individually and in coalition, seek rents from consultants or their agents. The
described mechanisms for detecting biases (11.2.8), and neutralizing outliers
(11.2.3) are useful as intermediate steps to shed light over differences between
EC members, or to cut short controversies that could not be reconciled.
Nevertheless, they should not distract from the need of EC collaboration in
mutually explaining and reconciling their individual positions, to reduce
information gaps and reach agreements. Well planned and well conducted EC
joint sessions would lend transparency and effectivness to the process for the
benefit of consultants and the supervising authority, as discussed in Chapter 13.
11.3 The Scoring Rule
11.3.1 Summary
In QCBS the selection is made on the basis of a scoring rule that maps the two
components of a proposal – the technical and the financial proposal – into a
total score. Examining the scoring rule currently recommended in the
Guidelines (para 2.21), and in the Bank SRFP it will be shown that this scoring
rule has several undesirable properties that require consideration. In particular,
it
• Favors higher priced proposals;

• Confounds and complicates bidding;
• Unnecessarily complicates the calibration of the selection rule;
• Allows irrelevant proposals to change the selection; and
• Invites strategic manipulation in the form of “shill bidding”.
These characteristics would have limited impact if the financial proposals

showed little dispersion. However as indicated under 11.1.1 the sample statistic
shows that the dispersion of financial proposal prices of the sample
assignments is very high.
11.3.2 Excursus: The ideal scoring rule
Ideally, one should ask: what is the value added by a proposal, i.e. by how
much does the Client’s wealth increase in expectation if he adopts that
proposal. Denoting that value added by V, the ideal total score of that proposal
is defined as the difference between its value added and its price p:
62
S=V–p (1)
Evidently, choosing the proposal with the highest total score is equivalent to
choosing the proposal that promises to maximize the Client’s net wealth.29
Therefore, this scoring rule is an ideal scoring rule which should be used
whenever the Client is able to compute the value added of proposals.
Unfortunately, in many applications the economic value of a proposal is

intrinsically difficult to determine. Many consulting assignments are part of a
larger project and they have, if judged by themselves, no value. Their value
derives from the value of the entire combination of all components. If a project
is essential, the marginal value of each part is equal to the value of the whole.
Such complementarities explain why the value of a consulting assignment can
often not be determined. In these cases, the ideal scoring rule cannot be
applied.
This problem is pertinent in the procurement of professional services which are

typically an essential part of a combination of different components (e.g. the
design and subsequent construction of a public infrastructure project).
If the ideal scoring rule cannot be applied, one is compelled to use substitute
rules such as the one currently employed under QCBS. However, one should
always keep in mind that they are imperfect substitutes. Therefore, if one
knows that a high value is involved, even though that value cannot be reliably
estimated, one may be well advised to give up trading–off price and quality and
select on the basis of quality only.
11.3.3 The currently used scoring rule
Under QCBS a proposal is characterized by (p,St): the price quoted, p, and the
technical score, St, determined by the evaluation committee. Proposals that fall
short of the stipulated MTS, Stmin, are excluded.
The scoring rule computes the total score, S, according to the formula:
S = αSt + (1 – α)Sf (2)
There, St is defined relative to “best practice” (a best practice proposal would

be given the maximum technical score equal to 100).30 Therefore, St is a
29
This scoring rule is assumed in the economics literature on scoring auctions (see, for
example, Che (1993) and Asker and Cantillon (2004).
30
In goods and services, for example in the World Bank IT Procurement, the score is
sometimes divided by the highest technical score.
63
percentage point, for example St = 80% . S f is defined relative to the lowest

price, pmin , quoted by all proposals that meet the quality threshold:
pmin
S f := 100.
p
(3)
The parameter α ∈[0,1] is the stated relative weight of quality.
QBS, LCS, and FBS can be viewed as special cases of QCBS by choosing α =
1 (100%) for QBS and FBS, and α = 0 (0%) for LCS, respectively.
11.3.4 The currently used scoring rule is flawed
The currently employed scoring rule has several undesirable properties that
should be corrected. The drawbacks are caused by the formula for computing
the financial score, equation (3). In the following these flaws are discussed and
based on the study findings, in the companion Chapter 18 and Annex 6 a
scoring rule is designed that remedies these defects at zero cost.
11.3.5 It favors higher priced proposals
In order to see the “hidden” discrimination in favor of higher priced proposals,

consider a particular proposal, say St = 70, p = 1,000. Now look at other
proposals with higher prices, say p ∈ {1,250, 1,500, 1750, 2,000, 2,250}. By
how much does one need to raise the associated technical scores in these other
proposals in order to reach the same total score (S=76) as the first proposal (St
= 70, p = 1,000).
Table 22 summarizes the answers, and Figure 6 plots them for a finer grid of
prices. Thereby, α is assumed to be equal to 0.8 (=80%). The technical score
that keeps the total score unchanged is called “equivalent technical score”.
As one can readily see, the currently used scoring formula requires higher
priced proposals to raise the technical score at a considerably diminishing rate.
As a result, a strong consultant who can reach a very high technical score can
win with a considerably high price against the proposal (St = 70, p = 1,000).
64
Table 22: (St, p) combinations that reach the same total score S
Proposal Given price p Equivalent technical score St

1 $1,000 70.00
2 $1,250 75.00
3 $1,500 78.33
4 $1,750 80.71
5 $2,000 82.50
6 $2,250 83.89
Figure 6: (St, p) combinations that reach the same total score
85
80
Technical Score
75
70
65
60
1000 1250 1500 1750 2000 2250
Price
Figure 6 displays the combinations of St and prices that reach the same total
score.
Since the technical score is a percentage point, every change in technical score,
dSt , is a relative change. Therefore, one may wish to measure price also in
such a way that a price change is a percentage change. This is achieved by
plotting St together with the Log of prices. Figure 7 displays the combinations
of St and Log( p ) that reach the same total score.
65
Figure 7: (St, Ln(p) combinations that reach the same total score
Relationship between St and Ln(p)
100.00
80.00
60.00
St
Series1
40.00
20.00
0.00
6.00 6.50 7.00 7.50 8.00 8.50 9.00 9.50
Ln(p)
If the relationship between St and Ln( p ) displayed in Figure 7 were linear, the
consultant could trade a percentage point price change for a change in technical
score at a fixed rate. However, under the currently used scoring rule that
relationship is not linear; instead, as Ln( p ) is increased, St goes up at a
diminishing rate. Therefore, if one compares a high and a low price proposal,
the high price proposal can compensate a given percentage point price hike
with a smaller increase in technical score. This shows how the discrimination
in favor of high prices shows up even if one looks at relative price changes.
Therefore, if one compares a high with a low price proposal, the higher price
proposal can compensate a given percentage point price hike with a smaller
increase in technical score.
In the following the “marginal reward for quality” (MRQ) is measured by the
slope of the curve displayed in Figure 7.31 A detailed technical definition and
analysis of the MRQ is in eq. (4).
11.3.6 It complicates bidding
A related flaw of the financial scoring rule (3) is that it raises the complexity of
consultants’ bidding problem.
Consultants differ in several dimensions: in their unit cost of staff, available

staff qualifications, country specific experience, and presence of
complementary tasks, to name a few. Given these constraints and the expected
reward for quality, each consultant chooses in his proposal, his own best
quality–price combination. This is already difficult because the behavior of
31
To be exact, it is the slope of that curve multiplied by 100 (see eq. (4) below).
66
evaluators is intrinsically difficult to predict and because a good choice also

requires a good prediction of rivals’ choices.
The currently employed scoring rule makes this choice even more difficult.
This is due to the fact that the MRQ depends not only on the individual
consultant own price, but also on the lowest of all prices. Predicting the reward
for quality is thus subject to uncertainty concerning rivals’ choices and
underlying characteristics. And making a good choice requires a good
prediction of rivals’ price strategies, which is exceedingly difficult to achieve
especially when technical proposals are likely to differ from each other. This
problem is particularly serious when the consultant short list is not
homogeneous or “mixed”, as it often happens in Bank funded assignments.
11.3.7 It unnecessarily complicates calibration
The complexity of the consultants’ bidding problem makes it also difficult for
the Client to calibrate the scoring formula to the specific conditions of his
selection case. There are two parameters that the Client needs to be calibrated:
the stated weight of quality, α, and the minimum technical score, Stmin.
In the vast majority of the sample cases Clients choose the same α and Stmin ∈
{80 and 70}. They also seem to choose α and Stmin independent of each other,
which is hardly optimal under all circumstances.
Table 23: Relationship between Stmin and a in QCBS

Correlation
Mean Stmin Mean α
coefficient
Highly Complex
73.43 0.78 –0.08
QCBS
Even if one looks at the subset of assignments with a high complexity grade,
where one would expect that Stmin and α should be relatively high, one finds
that both are below average and both are pretty much uncorrelated (see Table
23). This suggests that Clients do not pay much attention to calibration or
perhaps lack expertise or guidance to perform this task.
When preparing the RFP, some Clients recognize that, provided technical
scores and prices are positively correlated, the scoring formula tends to favour
quality by more than what the chosen stated weight of quality, α, suggests.
This distortion increases with increasing price dispersion p / pmin . In order to
mitigate this bias, Clients sometimes artificially lower α. However, this
67
mitigates one distortion by potentially aggravating another.32 Worse of all

many clients may avoid the problem altogether by assigning similar scores to
good and bad proposals; the low dispersion of technical scores across the
sample points in this direction.
Therefore, under the present formula, it is difficult, if not impossible for the
Client to calibrate the weight for quality in accordance to the characteristics of
the assignment.
11.3.8 It allows “irrelevant proposals” to change the selection
The financial score Sf depends both on a consultant’s own financial proposal

and the lowest price quoted by a shortlisted consultant who passes the
minimum technical score requirement, Stmin. This aggravates the distortions
already observed under 11.3.5-7, and it may give irrelevant proposals, that
stand no chance to win, a decisive influence on the selection of the winning
proposal. In addition, it may induce dishonest behavior in the form of “shill
bidding”. In the following these points are explained with two sets of
examples.
The first set of examples, summarized in Table 24 and Table 25, illustrates how
a very low priced proposal adversely affects the score of high price proposals.
This example assumes α = 0.8 and Stmin = 70.
The first table, Table 24, states a collection of scores and prices with a
relatively high spread.33 Evidently, in this case, consultant 5 wins the contract
with a technical score of 91.82 (the shaded row indicates the winning
proposal).34
32
Note, one may try to mitigate this by artificially lowering α. This works if the price
dispersion turns out to be high; however, if it happens to be low, it leads to an insufficiently
low weight of quality. This indicates that the artificially lowering of α is sometimes an
effective remedy, but at other times creates a new problem. Hence, it is a “cure” that may be
worse than the disease.
33
The numbers are not representative, but the sample includes cases with a similar spread of
technical scores and prices.
34
All examples are designed to illustrate a point; they are not claimed to be representative.
However, the spread of technical scores and prices is consistent with what we observe in
several cases of our sample.
68
Table 24: Outcome without strategic manipulation
Proposal Technical Score St Price p Total Score S

1 71.49 3,200,725 77.19
2 93.06 3,736,766 91.58
3 83.25 3,947,468 82.82
4 85.68 6,130,809 78.98
5 90.48 3,292,725 91.82
Now suppose consultant 1, who made an obviously irrelevant proposal, lowers

his price from $3,200.725 to the rock bottom level $2,311,316 (both numbers
are indicated in bold numbers). Then, the scores change to the levels stated in
Table 25. Evidently, proposal 1 remains irrelevant. However, as a result of this
change, consultant 2 now wins (the shaded row indicates the winning proposal)
without having changed neither his technical nor his financial proposal.
Hence, a very low priced and completely irrelevant proposal, that stands no
chance of being selected, may distort the selection.
Table 25: Outcome changed by an “irrelevant proposal”
Proposal Technical Score St Price p Total Score S

1 71.49 2,311,316 77.19
2 93.06 3,736,766 86.82
3 83.25 3,947,468 78.31
4 85.68 6,130,809 76.08
5 90.48 3,292,725 86.42
Of course, a good scoring rule should not have the property that an irrelevant
low price proposal has the power to change the selection in favor of a given
high price proposal. In a well designed scoring rule, the lowest price proposal
should not affect the scores of the other proposals.
This undesirable property of the currently used scoring rule may induce
strategic manipulation in the form of “shill-bidding”. The incentive to shill-
bidding can be shown simply by reinterpreting the above example.
69
Consider Table 24 as the result of consultants’ proposals before they engage in

strategic manipulation. Now suppose consultant 2 invites consultant 1 to act as
“shill-bidder” on his behalf.35 He instructs him to lower his price from
$3,200,725 to the rock bottom level $2,311,316. Then the scores change to the
ones stated in Table 25. As a result, consultant 2 wins, although he has changed
neither his technical nor his financial proposal, and only an irrelevant proposal
has changed.
Of course, a well-designed scoring rule should be free from such flaws.
11.3.9 The marginal reward for quality (MRQ)
The QCBS scoring rule applies a fixed relative weight of quality defined by the
parameter, α . However, the stated weight differs from its effective weight. In
the following, the effective weight of the technical proposal that is implicit in
the current scoring rule is computed.
The effective weight of the technical proposal is the marginal reward for
quality (MRQ) between quality and price. Loosely speaking, the MRQ defines
at what rate a consultant can trade quality for price without changing the total
score of his proposal.
In the present framework one has no absolute measure of quality and technical
proposals are scored only relative to best practice, as a percentage point.
Therefore, the MRQ is defined as a measure that says by how many percentage
points a consultant can raise his price, dp/p100, per unit percentage point
increase in technical score, dSt , without affecting the total score of his
proposal:
(dp/ p )100
MRQ :=
dSt S = const . (4)
Remark 1: While it is obvious that the ratio (dp/p)100 is a relative

(percentage point) price change, one may wonder why we describe dSt as a
relative (percentage point) change in technical score. The reason is that St is
already a percentage point. The technical score is computed relative to best
practice. A best practice proposal gets the technical score 100%; all other
proposals are scored as a percentage point of the best practice proposal. It
follows that if the technical score is increased from say St = 80 to S't = 85, that
35
A “shill bidder” is a bidder who acts on behalf of another bidder as a decoy. In the present
framework, a shill bidder quotes a rock-bottom priced proposal, together with a low technical
score on behalf of a consultant who quotes a high price/high quality proposal.
70
change, dSt = 5, is a 5% increase in technical score.
Straightforward computation gives:
α p α
MRQ = ≥ .
1 − α pmin 1 − α
(5)
For example, if α = 0.8, the MRQ is equal to 4.
Proof of (5): Totally differentiate equations (2) and (3) setting dS=0 (the total
score is kept unchanged). Then, one obtains:
0 = α dSt + (1 − α )dS f
(6)
pmin
dS f = − dp100
p2
p dp
= − min 100 (7)
p p
Combine equations (6) and (7) to compute the MRQ, defined in (4), and one
finds the asserted (5)
As one can see immediately, the MRQ is always greater than α/(1 – α), except
for the consultant who quoted the lowest price pmin.
71
MRQ
α
1−α
p'min pmin p
Figure 8: Relationship between the MRQ and price p and pmin
The relationship between the MRQ and the prices p and pmin is illustrated in
Figure 8. Evidently, the MRQ is a linear and increasing function of p; and a
nonlinear and decreasing function of pmin.
This shows clearly how the currently used scoring rule bizarrely favors high
price proposals by awarding them a higher marginal reward for quality. It also
shows how a reduction in pmin , say from pmin to p’min, raises the marginal
reward for quality of a given high price proposal. The latter explains why
consultant may gain from engaging a shill bidder, as indicated already by
example in the previous section.
While the MRQ is increasing in p and decreasing in pmin, which is undesirable,

it is independent of the technical score St, which may also be undesirable.
Indeed, under some circumstances one may not wish to reduce the marginal
benefit for quality as one approaches best practice.
11.3.10 Are these flaws empirically significant?
The discrimination in favor of higher priced proposals does not have much
impact if prices are very close. Therefore, one may ask: does one really observe
highly dispersed prices for a competed assignment? The answer is yes; more
than one might expect.
72
In the study sample one observes a wide dispersion of prices, and thus of
MRQ’s, within assignments. The MRQ ranges on average between the number
3.5 (for the lowest price proposal) and 7.33 (for the highest price proposal). In
other words, the marginal reward for quality of the highest price proposal is, on
average, more than 1.8 times as high as that for the lowest price proposal.
It is also instructive to look at the dispersion of the ratio of the highest to

lowest price, pmax/pmin. If that ratio is greater than 2, that means that the highest
price quoted in a given assignment is more than twice as high as the lowest.
Figure 9 plots the cumulative frequency distribution of pmax/pmin. Evidently,
there is significant frequency mass on price ratios above 2, even above 3.
Indeed, the highest observed ratio in the sample is 5.5, and a ratio greater than
2 is observed in 35% of all QCBS assignments.
Figure 9: Cumulative frequency distribution of pmax/pmin
This suggests that the “hidden” discrimination in favor of higher priced

proposals is indeed significant. This is particularly disturbing since one
observes no significantly positive correlation between price and quality.
If the correlation between price and quality, and between technical and
financial scores were consistently positive and significant, the favoring of
higher prices would at least favor high quality. However, on average, the
correlation coefficient between technical and financial proposals, ρ, is
negligibly small (ρ = 0.09), although positive. Moreover, in almost half the
assignments that correlation coefficient is negative, i.e. high prices tend to be
the associated with low qualities (see Table 26). This picture improves only
slightly if one separates the populations of proposals from Part I and Part II
countries.
73
Table 26: Correlation between technical/ financial scores

Incidents of Incidents of
Correl. Mean ρ Stdev
ρ<0 ρ>0
Coeff. 0.09 0.76 47% 53%
11.3.11 Conclusions
The currently used QCBS scoring rule discriminates in favor of higher priced
proposals particularly when the price dispersion is high. This is reflected by the
fact that the marginal reward for quality is an increasing function of the price
p36. In addition, the MRQ depends also upon the lowest price pmin in a way
which not only aggravates the resulting distortion but also makes it difficult for
consultants to predict how the rule rewards them for quality. The latter makes it
also exceedingly difficult for the Client to calibrate the scoring rule to the
specifics of the procurement case.
This discrimination in favor of higher prices would not be significant if prices

were similar. Also, it would perhaps not be a major concern if price and quality
were positively correlated. However, in the sample a high dispersion of prices
is found (and a low dispersion of quality scores). And no significant correlation
between price and quality was found. Indeed, the correlation between price and
quality is almost as often negative as it is positive. It seems that clients avoid
the problem posed by the formula by assigning similar scores to good and bad
technical proposals; the low dispersion of technical scores across the sample
points in this direction.
In order to remedy the observed flaws of the currently used scoring rule, one
needs a rule that meets the following requirements:
1. The marginal reward for quality (MRQ) offered to a consultant should

be independent of the price he quotes in his proposal; in other words, it
should not be a function of the price quoted in a proposal.
2. The marginal reward for quality (MRQ) offered to a consultant should
be independent of the prices quoted by other consultants, in other
words, it should not be function of the prices quoted in other proposals.
3. The scoring rule should perhaps be flexible and permit that the marginal
reward for quality depends on the technical score. (The latter is relevant
36
One could argue that the statement is not supported by facts. By scoring proposal quality
very closely, as shown under 11.1, in practice evaluators more than neutralize the favor that the
formula bestows on high quality-priced proposals; the distorting formula generates another
distortion in the opposite direction (Assuming the proposal quality and proposal prices are
positively correlated).
74
if the Client considers an increase in quality as less and less valuable as

a proposal comes closer to best practice.)
The new scoring rule proposed in the companion Chapter 18 and Annex 6 may
bring us closer to these goals.
Under the present scoring rule borrowers should be advised to use QCBS not
by default but cautiously i.e. when proposals are comparable, assignments are
likely to be executed in essentially the same way by the consultants. These
consultants should be organizations of like type and business. Where this is not
possible the most sensible way to go is not to relate quality and price upfront
but consider selection methods where prices are not compared upfront, such
QBS, FBS or CQS.
11.4 The Contract
Based on the total population of FY03-04 prior review contracts37, the average
World Bank funded consultant contract is $ 891,000 (median $ 432,000)38 with
a mean duration of 26.5 calendar months. The average international consultant
staff month remuneration rate of $18,400 (median $ 16,700), the corresponding
national consultants rate $ 4,900 (median $ 3,100).
In the study sample, the contracts are evenly split as 50% are time-based and
50% lump-sum. According to paragraph 4.2 of the Guidelines, time-based
contracts should be used when objective scope and/or duration of the
assignment are difficult to determine in advance. Lump-sum contracts, should
be used when content, duration, output and contract payments can be clearly
defined in advance.
Bank funded assignments very often require consultants to be hired on time

basis because of the duration and the complex circumstances surrounding
Bank-funded projects. Often, even relatively simple assignments should be
treated with caution and can turn out to be risky for consultants because of the
Clients’ difficult administrative context and low capacity levels. Surprisingly
no significant correlation is found in the sample between the type of contract
and the complexity of the assignment. Actually, only 44% of the complex
assignment (ToR graded 10-12) turn out to have a time-based contract.
Lack of risk analysis emerges here similar to what was observed for the other
decisions regarding the ToR and the choice of the selection method. This
explains the observed inefficient distribution of risks between Clients and
37
The prior review contracts awarded in FY03-04 were 1972; contracts awarded to UN
organizations are not excluded.
75
consultants that eventually negatively affects the quality of the services, and
the success of the project. Also the integrity of the contractual relationship is at
risk as the weaker party tries to make good his losses in spite of the contract.
It may be interesting that of the sample contracts, 52.6% were awarded to Part I
consultants.
Table 27: Contracts Awarded to Part I and Part II Country Consulting Firms
(Study Sample of 114 Assignments)
USA 12 Bangladesh 4 Bangladesh 2
Germany 12 Brazil 3 South Africa 2
Australia 6 Pakistan 3 India 1
France 5 Indonesia 3 Thailand 1
UK 5 Zambia 3 Lesotho 1
Canada 4 Uganda 2 Peru 1
Sweden 3 Lebanon 2 Romania 1
Netherlands 3 Bolivia 2 Costa Rica 1
Denmark 2 Afghanistan 2 Ghana 1
Japan 2 Ghana 2
Spain 1 Sierra Leone 2
Belgium 1 Turkey 1
Finland 1 Russia 1
Israel 1 Belarus 1
Hong Kong 1 Slovakia 1
Austria 1 Egypt 1
Others - Others 10 Others -
Total 60 Total 43 Total 11
% 52.6 37.4 10
12 National Consultants Participation

For a long time now39 Bank emphasizes promoting national consultancies so
that clients can develop the capability to prepare, design and carry out their
own development work, and also on expressing a “regional preference” for Part
II consultants seeking work from other Bank’s clients. The most important
steps to this effect, were the recognition of the importance of cost as a
consideration in the 1981 Guidelines, and in 1997 of “Bank’s interest in
encouraging the development and use of national consultants in its
development member countries” as the fourth consideration of the Guidelines.
This consideration is supported by the following preferential provisions:

39
The 1966 Guidelines para 3.1 page 2 for the first time indicate explicitly that” The
employment of domestic firms is encouraged where such firms are found to be qualified, either
alone or in combination with foreign firms”.
76
• Para 2.6: “the short list shall comprise (…) no more than two firms from
anyone country or at least one firm from a developing country unless
qualified firms from developing countries are not identified”;
• Para 2.7: “The shortlist may comprise entirely national consultants (firms
registered or incorporated in the country), if the assignment is below the
threshold established in the procurement plan”;
• Para 2.15: the criteria of evaluation allow for up to ten points to be
allocated to proposals with “participation of nationals” as key staff;
• Para 2.7 allows subsidiaries of international consultants registered or
incorporated in the country the status of national consultants.
Most important, the introduction of QCBS, FBS and LCS in 1997, has made
explicit the comparative cost advantage of Part II consultants.
12.1 Part II Consultants participation rates
Figure 12 summarizes the participation rates of consultants from Part II

countries as lead firms in the various stages of the selection process of the
assignments included in the study sample.
Figure 10: Participation Rates of Part II Country Consultants as Lead Firms
Mean Median
EOI
42% Shortlist 33%
43% Proposals 33%
42% Evaluated 33%

Proposals
47% Awarded
Proposal
77
In addition, the study sample indicates that in 59% of the cases lead consultants
from a Part I countries propose their services in association with one or more
consultants from a Part II country, as their junior partners in the joint venture or
as subconsultants.
Over time the share of Bank funded prior review contracts in $ terms awarded
to Part II country consultants have markedly increased, as shown Table 28.
Table 28: Prior Review Consulting Contract $ Value Distribution in %

Country of Origin
FY 03 - 04 43% 49% 8%
FY 95 58% 33% 9%
FY 72 - 76 81% 15% 4%
Part II consultants have increased their participation in Bank projects in their

own country, at least quantitatively. Table 29 illustrates Part II consultants’
participation in the sample contracts in terms of staff months and remuneration
excluding reimbursable items. Their participation decreases with increasing
complexity of the assignment ToR. This should not come as a surprise
considering that national consultants’ participation has been moving from the
less risky and simple to the more complex assignments.
Table 29: Participation Rates of Part II Consultants in Sample Contracts by ToR

Grade, Staff Months and Remuneration Share
Staff Months Remuneration
Grade 12 -10 41% 33%
Grade 9 - 7 58% 44%
Grade 6-4 89% 78%
Overall 59% 48%
As indicated in Table 28, Part II consultants have been less successful outside
their own country. The relatively few cases observed consist of Part II
subsidiaries of multinational accounting firms and IT consultants or NGOs. For
Part II consulting engineers, economists and other professionals, exporting
services to other Part II countries remains very difficult, especially to public
sector clients.
78
12.2 Creation of National Consultancies as Development Objective
Even if data available suggest that the Guidelines provisions may have
contributed to increased participation of national consultants, probably because
of the relatively small weight of its lending, the sustainable development on
national consulting industries has not been achieved. Growth of independent
consulting firms in Part II countries remains slow and qualitatively uneven as
their economies assign limited roles to the professional services sector. Apart
from weak private demand for professional consulting services, and weak
organization of the professional consulting and expert services industry, public
sector attitude can be characterized as at best indifferent to the problems of the
industry. The reasons that can be placed on government side for slowing down
sustainable growth are:
Weak Demand: Public sector demand for consulting services is narrow and
unstable. Many public sector agencies consider that there is little that
consulting services can contribute in terms of innovation, operational
efficiency or cost. The value of impartial advice in formulating effective
solutions and transparent choices is ignored, or dismissed by bureaucracy and
government. Most clients also lack a vision of the role that professional
consulting services can play for their country as engines of the knowledge
based economy.
Poor Regulation Crowds out Quality: public procurement of Clients’ countries
regulates consulting services similar to goods or at best, to simple technology
services. As it is a difficult to rank differences in observed quality of
professional services, responsible officials score consultants technical
proposals closely to each other (see 11.1.) and award the contract to the lowest
bidder. The consequence of this manner of awarding consulting contracts is a
downward quality spiral that crowds-out the better consultants from
government contracts and condemns consulting businesses to fail or to bare
professional subsistence.
Government Consultants: in most countries of the ECA, and some of the SAR
(Pakistan) and EAP (Viet Nam) regions, poor regulation is compounded with
resort, by the public administration, to in-house technical departments, or
affiliated public institutes for technical and design services. While this practice
may be the response to the paucity of consulting services in the local market, it
contributes to institutional failure by eliminating the independent, impartial
analysis and supervision that are often vital to success. Predatory practices by
government consultants and tied technical aid can further aggravate the plight
of national private professional consultants and experts.
Because of the above factors, it is not surprising that the data from the national
consultant associations, confirmed in Bank surveys conducted in FY 03-05 in
Colombia, Pakistan, Turkey and Viet Nam, indicate that the consulting services
79
sectors of these countries stagnate, or cannot take off as independent, private

organizations, while international consultants operate only occasionally in most
of these countries to render their services to ODA programs or to
multinationals.
13 Transparency
In consultants’ selection, transparency should enable the three main parties
involved, i.e. the Client, the EC, and the consultants, to observe the fair
application of the rules and respect for the procedures that should lead to the
efficient and fair contract allocation. When information is asymmetric, lack of
transparency can raise the parties’ concern that for non-compliance with the
“rules of the game” by any other party, their individual interests are not fairly
protected. This weakens in some the confidence in the process and their
commitment to the rules of the game, and constitutes for others a deterrent to
participation.
Since 1997, the Guidelines list transparency as one of the main considerations
and provisions are in place that must be complied with and may be verified by
the parties involved for their fair treatment:
• Publication of the Request for Expressions of Interest;
• Issuance of the same RFP to all short listed consultants;
• Disclosure of the same information available to short listed consultants;
• Disclosure of all evaluation parameters (criteria, MDS, quality weight);
• Invitation for short listed consultants to joint pre-proposal conference;
• Disclosure of all Client’s responses to consultants’ questions;
• Provision of sufficient time for delivery of proposals;
• Submission of all proposals before a specified date;
• Return of sealed late proposals;
• Public opening of financial proposal envelopes;
• Preparation evaluation reports using standard formats;
• Publication of the evaluation scores; and
• Optional individual debriefings for unsuccessful consultants.
Many of these provisions are inspired from procurement for goods, and are
meant to prevent open participant discrimination. The question about these
safeguards is to what extent they bring tangible benefits for Clients, evaluators
and consultants, considering that some raise process cost above what should be
the reasonable norm in a consultant selection, and others put at risk its
efficiency and even effectiveness e.g. the disqualification of substantially
responsive proposals on formal grounds or because of marginal discrepancies
from the provisions.
80
There is no mention in the Guidelines of what lends transparency to the actual

evaluation process and its subsequent decisions, and of how transparency
relates functionally to efficient quality discovery, confidentiality and credibility
of the selection process. As it is not an objective in itself, transparency is best
explained by the purpose it accomplishes namely the demonstrably fair
selection of the best proposal by clearly explaining the rules of the game and
showing that these are respected by the parties involved.
The client (the principal) does not stand in direct relationship with the
consultants and seeks transparency to satisfy himself that the EC (the agent)
operates in the client’s best interest and not in his own interest. The objective
of transparency in this context is essentially the client’s control over the EC.
The EC seeks transparency to secure adequate information from and avoid

being misled by consultants whose immediate objective is to make a sale.
Consultants have an incentive to act transparently (reveal all information
required) and honestly if they in turn have confidence in the manner in which
their proposals are evaluated and the selection process conducted. Without such
confidence consultants may either abandon the competition or decide not to
play by the rules established in the RFP.
There is no mention in the Guidelines of what constitutes transparency of the

evaluation process and how it relates to quality, to fair competition and to fraud
or corruption. Paras 1.22 and 1.23 merely explain how clients and consultants
can incur sanctions for acts of fraud or corruption that they could commit when
transparency is missing.
Transparency of the evaluation process or its perception for the parties

involved is brought by:
i) Clarity and comprehensiveness of Guidelines and RFP;
ii) Protocols and rules governing EC sessions, evaluation and award decisions,
client delegation, and EC protection from external interference;
iii) Commitment of EC members, clients and consultant to the RFP rules;
Weaknesses in these areas affect transparency for any on the three parties,
offering excuses to the affected to depart from the rules.
13.1 Clarity and Comprehensiveness of Guidelines and RFP
The Guidelines include a few of provisions that, for the manner in which they
have been worded and occasionally grafted onto the existing text, can confuse
users as of the actual guidance being given. They can lead to sub-optimal
decisions by those, less experienced, in charge of RFP preparation with regard
to important issues such as the choice of the selection method. They also can
81
render advice by Bank staff less precise. Some of these imprecision are listed,
commented and the proposed rectified text laid out under Section 16.1.
13.2 Regulating Evaluation and Selection
The evaluation process and its results are often criticized by consultants and
supervising authorities as opaque and questionable for the manner in which
they are staged or conducted. As discussed in Chapter 11, the duration of the
process and the dispersion of technical scores across EC members suggest that
evaluators do not have protocols that they must follow to discuss evaluations,
fill information gaps, search for agreement and decide.
13.3 Commitment of EC, Client and Consultants
The EC’s decisions based on objective factors and sound judgment should
determine the outcome of the evaluation. In the study sample, EC members are
often appointed on the basis of rank in the executing agency and/or supervising
ministry (see Table 14), instead of professional capacity and commitment to
the rules of the game and on an effective selection process.
Lack of commitment from the client or the EC or both, can encourage

consultants to give up their own commitment to the rules of the RFP, for
example by including untrue information in their proposals, or trying to
influence the selection with pressures on the EC or the Clients.
Anecdotes from consultants and Bank staff confirm that the factors explained
in 13.1, and 13.2, easily raise the consultants’ suspicion of poor or unfair
evaluations. The lack of RFP clarity and of reliability of the evaluation animate
unethical consultants, EC members and clients to seek redress in corrupt or
fraudulent practices, while the ethical consultants will give up participation.
14 Warning Signs
The review of the selection process conducted up to this point and the data
analysis detect some disturbing patterns:
Inefficient and expensive selection process
The duration, the bureaucratic overload and the cost of the process have
become excessive for all parties involved as process compliance is given more
importance than process results.
Procurement process ignores project risk, complexity of ToR

82
Clients apply rigidly the provisions of the guidelines but ignore the risk
characteristics of an assignment’s ToR when planning the selection process. In
91% of all competed prior-review contracts, Clients choose QCBS without
regard for the complexity of the assignment ToR and the value of its project, or
the homogeneity of the short list. Clients’ preference for QCBS may be
influenced by the wording of certain paragraphs of the Guidelines.
Low dispersion of quality scores and high dispersion of prices
Combining the two average dispersion coefficients for quality 1.12 and for
price 1.79, it shows that Clients declare equally fit for purpose proposals with
very different prices, even more so if assignments are complex, very complex
or refer to a valuable project.
No apparent relationship between price and quality
In the sample data no significant positive correlation between price and quality
is found. To the contrary, that correlation coefficient is almost as frequently
negative as it is positive.
Award distribution suggests Clients prefer low-priced proposals
Price considerations prevail in the selection. Although the highest technical

score is awarded the contract in 64% of the cases, the lowest priced proposal is
awarded the contract in 67% of them. The lowest priced is also the highest
technical proposal in 35% of the cases.
QCBS’s scoring formula is flawed
The currently used QCBS scoring rule has several undesirable characteristics
that render the bidding process problematic for clients and consultants as well
as likely to distort the behavior of both.
Evaluation Committees are often unsuited for the task
Even when honest, evaluation committee members often lack the impartiality
and the experience to evaluate complex technical proposals.
Transparency is treated as an end not as a mean
Clients understand transparency provisions as a bureaucratic end in itself and

not as a tool to provide symmetric information, improve competition, and
outcomes.
83
Part IV
Recommendations
84
IV Recommendations
The previous chapters have called attention to the need of a regulation that
reduces process complexity and cost, attracts capable consultants, and fosters
the development of consulting capacity in Clients’ countries. The study has
also shown that the two-stage procurement model, subject to the Guidelines’
six main considerations, constitutes a stable procurement framework. This,
however, requires to be effective and transparent, a coherent and precise
formulation of the Guidelines’ text.
Important policy changes, such as the considerations for cost introduced in

1981 and for transparency in 1997, have required extensive modifications and
addition of several provisions. These and several other occasional changes
affect the clarity of the present Guidelines text with respect to their intended
purpose, stretch the adequacy of the RFP as a search mechanism, and impair
the reliability of the selection process.
The warning signs of the previous chapters call attention to the need to make
the Guidelines a simpler decision tool that can lead its user to the correct
analysis and decision on the particular selection case, rather than to a set of
prescriptions to be rigidly complied with. The recommended text adjustments
will guide clients to a more principled and coherent application of the different
provisions particularly on the choice of the selection method and the fairness of
the short list. The study distinguishes between what can be achieved in the
short term by explaining better the present policy so that it can be correctly
applied, and what should be considered in order to bring the guidance closer to
emerging professional practices, to the Bank evolving project portfolio, and to
encouraging more effectively formation of national consulting capacities of
high quality.
The study recommendations’ part follows the same pattern of the findings’
part, by covering strictly the Guidelines considerations on the selection
process. The needs of users’ capacity building and integrity equally crucial to
the outcome of the selection process are mentioned whenever needed, but are
not specific subjects of this study. The role of Bank staff in the supervision of
the selection process and the quality of assistance to the client are also not
discussed, although they require attention in the nearest future.
85
15 Raise Efficiency and Economy

Limiting the duration of the selection and its cost by reducing the complexity
and volume of documents to be prepared and evaluated, as well as the related
bureaucratic complexities should be the first priority. Both clients and
consultants would benefit from them.
The suggested measures to raise efficiency and economy are compatible with
the 2004 Guidelines, and arise from studying the sample, from interviews with
Bank staff and consulting firms, and consist:
• Stemming assignment fragmentation and compulsive tendering
• Reducing Expressions of Interest workload
• Using simplified selection procedures whenever suitable.
15.1 Bundle Assignments and Allow Continuation
Project procurement plans often list small and closely related consulting
services assignments that are tendered separately for no obvious reason (see
Annex 9). Bundling and continuation of compatible assignments should be
encouraged.
• Bundling multiple tasks in one single assignment ToR whenever

technically feasible and administratively reasonable;
• For staged assignments, always contemplate in the ToR and the RFP the
possibility for the continuation of planned services by the same consultant
if he has performed as promised in the contract.
These measures would require decisions and extra planning during the
preparation of the project by clients and TTL’s but their efforts would be amply
compensated during implementation in terms of accountability, coverage,
coherence and lower cost of consultant’s advice. Aggregation will also
facilitate supervision by clients and the Bank.
The same applies to staged assignments. Longer and larger contracts will
reduce interruptions and contract administration overloads. Both measures
would also attract the participation of qualified consultants seeking longer and
stable client-consultant partnership.
15.2 Rationalize Recourse to Request for Expressions of Interest
The request for EoI, while satisfying the need of wide and transparent diffusion
of opportunities, can be time consuming and costly. This task is particularly
challenging for less experienced clients who lack familiarity with consultants’
86
selections, and spend considerable time examining unfamiliar information and

preparing short-listing reports. The same exercise can be a nuisance for clients
that already have sufficient experience with the identification and the
employment of consulting resources.
The following measures are aimed at reducing the number and simplifying the
searches of consultants for short listing by:
• Limiting the request for EoI to only one comprehensive event, namely at
the moment project preparation is started, without repeating it, for every
assignment. Consultants could express interest for more than one
assignment under one EoI.
• Inviting consultants to provide only their standard brochure, and if
necessary a “limited” number of examples of their choice, of recent
significant experience related to the proposed assignment. This will force
the consultants to perform the prior screening of their qualifications and
avoid swamping clients with information. Strictly abstaining from
requesting CVs, preliminary technical approaches and similar approaches
that belong only to the proposal phase.
• Allow clients to request consultants the presentation of a professional
liability insurance.
• Allowing clients to ask for a fee upon submission of their EoI. This would
induce consultants to be more discriminating or “sincere” in their
expressing interest. This could substantially reduce the number of EoI and
evaluation work to be conducted by clients.
• Asking clients to prepare a brief memo (not more than two pages)
explaining the reasons for inviting the short-listed consultants, not a lengthy
“Short Listing Report”.
• Exercising reasonable flexibility in the application of short-listing rules that
insist on geographic spread and number of short-listed firms when they do
not add value to the quality of competition.
• Encouraging creation of clients’ exchangeable databases on qualifications
of reputable consultants, including evaluations of past services.
• Strengthening capacity of Bank staff, in explaining the considerations that
need to be made in preparing a short list and provide upon request a list of
reputable consultants, rather than leaving clients exposed to the risk of poor
decisions.
87
15.3 Encourage the Use of Available Simple Selection Methods and

Simplified Procedures
The Guidelines offer different selection methods, and the standard RFP
features options allowing clients to limit time and the volume of information:
• Update the $ threshold for admitting the use of CQS by raising it, for
example to the present Bank-wide median contract value. This measure
could be adopted on a pilot basis.
• Recommend the use of RFP Simplified Technical Proposals (STP) option
as default for contracts of less than US$ 1,000,000, independent from the
chosen selection method (except CQS). (this measure would capture 78.5%
of all 2003/04 prior review contracts).
The benefits of expanding the use of CQS would be also qualitative, and will
be discussed in the next chapter. As for the STP It is suggested that consultants
be asked to submit an STP of not more than 20 pages (single side) and CVs of
not more than 2 pages for proposals with estimated cost of US$ 1 million or
less.
16 Raise Consideration for Quality

In Chapter 9 it was shown that the outcome of the selection process often
depends upon how correctly the client interprets and applies the Guidelines in
preparing the RFP. While strengthening clients’ capacity will require training
and time, in the meantime selection performance and outcomes can improve
by: a) clarifying the guidance, and b) leading clients to the use of simplified
procedures. In Chapter 9 it was shown that clients apply mechanically the
Guidelines, choose the selection method and set the other criteria and
parameters of the RFP in formal compliance with the requirements for fairness,
transparency but without much regard for the complexity of the ToR and the
effectiveness of the selection process.
16.1 Introducing the Consideration for Risk
The Guidelines would add value to clients’ decisions by bringing to their

attention the importance of taking into account the risk characteristics of the
assignment ToR when fixing the parameters of the selection process, in
particular the choice of the selection method (i.e. the weight of quality), the
type of contract and the short list. This could be done under para 2.3 of the
present edition of the Guidelines.
88
16.2 Avoiding Misapprehension: Rephrasing the Guidelines
Contrary to the spirit of offering ample applicability to a wide range of

services, users are led to a somewhat rigid use of the Guidelines, partly due to a
lack of clarity of a few provisions and others creating contradictions. These
were originated by occasionally drafting new procedures, or even policies, to
the original text without regard for compatibility and in doing so falling foul of
confusing or contradicting other existing provisions and policies.
The following comments and suggestions to specific paragraphs of the

Guidelines are intended to provide clearer guidance and to make the Guidelines
more precise, and consistent with their original objective and main
considerations.
16.2.1 Para. 1.5 (General Considerations)
Para 1.5 is far from providing a clear message. On the one hand, the first
sentence of para.15 states that the considerations laid down in para. 1.4 “... can
best be addressed through competition ... in which the selection is based on the
quality of the proposal and, where appropriate, on the costs of the services to
be provided.” On the other hand, Sentence 3 refers to QCBS as “... the most
commonly recommended ...” selection method.
In order to provide clear guidance, Sentence 3 of para.1.5 should be rephrased

as follows:
“The Bank considers that, in the majority of cases, these considerations can
be best addressed through competition among qualified short-listed firms in
which the selection is based on the quality of the proposal and, where
appropriate, on the cost of the services to be provided. Sections II and III of
these Guidelines describe the different methods of selection of consultants
accepted by the Bank and the circumstances in which they are appropriate.
Since the procedures for Quality- and Cost-Based Selection (QCBS) address
all potential stages of the selection process for all selection methods accepted
by the Bank, Section II of these Guidelines describes the procedures for
QCBS in detail. However, QCBS is not the appropriate method of selection
for all cases; therefore, Section III describes other methods of selection and
the circumstances in which they are more appropriate.”
16.2.2 Para. 2.1 (Adoption of QCBS)
The current para. 2.1 simply refers to a “competitive process” but fails to
address the precise requirements for adopting QCBS. One has to study para.
3.2 on QBS to understand the requisites for the adoption of QCBS. This
89
approach certainly does not contribute to a better understanding of the selection

process in general and the adoption of QCBS in particular.
Therefore, it is recommended to replace the first and second Sentences of para.

2.1 with the following ones:
“QCBS uses a competitive process among short-listed firms that takes into
account the quality of the proposal and the cost of the services in the
selection of the successful firm. Cost as a factor of selection shall be used
judiciously. QCBS is appropriate when the assignment meets the following
conditions:
(i) the consultant scope of work can be precisely defined;
(ii) the ToR are well specified and clear;
(iii) the risk of undesired downstream impacts is low and manageable;
(iv) the key staff time as well as the skills set required of the consultants can
be estimated with reasonable precision by the client; and
(v) comparison of the technical and financial aspects of the proposals is
likely to be simple.
In addition, the last sentence of para. 2.1 should be moved to para. 2.11
(Instructions to Consultants).
16.2.3 Para. 2.16 (Evaluation Criteria)
In para. 2.16, Sentence 2 states that for the evaluation of quality “subcriteria
under methodology might be innovation and level of detail”. The reference to
“innovation” is misleading since QBS not QCBS should be used since the
Client expects the consultants to demonstrate innovation in their proposals (see
para. 3.2, lit. (a)).
Accordingly, Sentence 2 of para. 2.16 should refer to the standard subcriteria
for the methodology indicated in the Data Sheet of the SRFP.
Therefore, it is recommended to rephrase Sentence 2 of para. 2.16 as follows:

“For example, subcriteria under methodology might be technical approach
and methodology, work plan, and organization and staffing.”
16.2.4 Paras. 2.16 and 2.17 (Evaluation Criteria “Methodology” and “Key
Personnel”)
Paras. 2.16 and 2.17 contradict each other. While Sentence 6 of para. 2.16
states that “more weight shall be given to the methodology in the case of more
complex assignments”, Sentence 2 of para. 2.17 lays down that “Since key
personnel ultimately determine the quality of performance, more weight shall
be assigned to this criterion if the proposed assignment is complex”.
90
Furthermore, the reference to “complex” assignments is misleading since QBS,

and not QCBS should apply to complex assignments (see para. 3.2, lit. (a)).
In order to provide clear guidance on the adoption of QCBS and the calibration
of the individual evaluation criteria, paras. 2.16 and 2.17 should be rephrased
as follows:
Sentence 6 of para. 2.16:
“More weight shall be given to the methodology (for example in the case of
complex assignments, or in the initial stages of a project where the
consultants are expected to further define the scope of work described by the
ToR)”.
Sentence 2 of para. 2.17:

“More weight shall be assigned to the key personnel if the scope of work is
precisely defined in the ToR and the main objective is to have the most
qualified experts.”
16.2.5 Para. 2.16 (Evaluation Criterion Transfer of Knowledge)
Footnote 24 of para. 2.16 merely states that more points might be assigned to
“transfer of knowledge” (ToK) if it is the main objective of a given assignment.
Considering the relevance of Capacity Development assignments in current

Bank-financed projects, the transfer of knowledge criterion should be
addressed in a new paragraph to be inserted after para. 2.17. In addition, to
avoid overlaps of the criteria “methodology” and “transfer of knowledge” one
should distinguish between assignments where transfer of knowledge is either
the main objective or merely a minor subcomponent.
Therefore, it is recommended to delete footnote 24, to insert a new paragraph

between 2.17 and 2.18 and to phrase it as follows:
New para. on transfer of knowledge:
“In cases where “capacity development or transfer of knowledge” is the main
objective of the assignment, no points shall be assigned to this criterion.
Points for the approach on “transfer of knowledge” shall be awarded
exclusively under the evaluation criterion “methodology” and its respective
subcriteria. If capacity building, transfer of knowledge or training are merely
a subcomponent of the assignment (for example a training course), 0 to 10
points shall be awarded under the transfer of knowledge criterion.”
91
16.2.6 Para. 2.23 (Combined Quality and Cost Evaluation) and para 2.11
(Instructions to Consultants)
Sentence 2 of Para. 2.23 states that “the weight for ‘cost’ shall be chosen,
taking into account the complexity of the assignment and the relative
importance of quality”. However, Sentence 3 invokes the impression that 20
points shall be the normal weight for ‘cost’ under QCBS. Furthermore, para.
2.23 should be only concerned with the Combined Quality and Cost
Evaluation.
In order to provide clear guidance it is recommended to rephrase Sentences 2

and 3 of para. 2.23 as follows, and to move them to para 2.11 (Instructions to
Consultants) as Sentence 1, as follows:
New Sentence 1 of para. 2.11:

“The ITC shall contain all necessary information that would help consultants
prepare responsive proposals, and shall bring as much transparency as
possible to the selection procedure by providing information on the
evaluation process and by indicating the evaluation criteria and factors and
their respective weights and the minimum technical score. The relative
weight to be given to the quality and cost shall be determined for each case
depending on the nature of the assignment. Cost as a factor shall be used
judiciously. Except for the type of services specified in Section III, the
weight for ‘cost’ shall be 20 points out of a total score of 100 for standard
assignments. For less simple and almost complex assignments only 10-15
points shall be assigned to ‘cost’ depending on the Client’s preferences.”
It is also recommended to rephrase Sentences 1, 4, and 5 of para. 2.23 as

follows:
“The total score shall be obtained by weighting the quality and cost scores
according to their relative weights laid down in the RFP and adding them.
The firm obtaining the highest score shall be invited for negotiations.”
16.2.7 Para. 3.2 (Adoption of QBS)
Para. 3.2 does not address all cases when QBS should be adopted. Infact one
has to refer to para. 2.8 to find out that QBS is applicable when a mixed
shortlist is used.
Therefore, it is recommended to add the following Sentence to para. 3.2:

“In addition, QBS or CQS (for small assignments) shall be used if the
shortlist does not comprise consultants of the same category, similar capacity,
and business objectives (see para. 2.8).”
92
16.2.8 Para. 3.5 (Adoption of FBS)
Para. 3.5 on the use of FSB, requires that the assignment is “simple, that it can
be precisely defined and that the budget is fixed”.
It is recommended to rephrase Sentence 1 of para. 3.5 as follows:

“FBS is appropriate only when
(i) the ToR are precisely defined;
(ii) the time and the consultant inputs can be accurately assessed; and/or
(iii) the budget for the assignment is fixed and cannot be exceeded.”
16.3 Reduce Burden of RFP Preparation and Proposal Evaluation
Issuing a complex RFP and asking for large proposals adds little to the
effectiveness of the outcome if the process itself is not properly conducted.
More so if the contract is small.
In the short term, without modifying the Guidelines, for the smaller
assignments with well specified and detailed ToR, it is recommended to use
selection methods that do not require the preparation of complex and lengthy
RFP’s that burden Clients with too complex and time consuming evaluations:
• For small contracts amounts estimated up to the present median contract
value of Bank funded contracts40 Clients should be allowed to adopt CQS;
• STP should be the rule for assignments up to $ 1,000,000 or 50 key staff
months;
Apart from the efficiency gains and cost reductions mentioned under para 15.3,
an important quality of CQS is of being based on past qualifications of the
firms, emphasizing the importance of consultants’ accumulated qualifications
and competencies. In the face of imperfectly observable (promised) quality, the
best measure of capability is a consultant’s past track record, which is easier to
verify than proposed methodologies and curricula vitae that once promised,
very often change before the assignment can start. Firms’ past qualifications
constitute more reliable information as they cannot be as easily tailored as
methodologies and CVs. Only one proposal is requested under CQS, which
will reduce process cost and shorten the time required for conducting
evaluations.
For contracts estimated between the present Bank median contract value of
US$ 432,000 and US$ 1,000,000 (similar to ADB41) the RFP should allow for
40
The current threshold is US$ 200,000 (see footnote 32 to para. 3.7 of the Guidelines).
41
ADB has set the default lower threshold for the STP at $600,000 and the upper at
$1,000,000.
93
the STP as the default recommended option. For contracts above US$
1,000,000 or 50 key staff months, the full technical proposal FTP should be the
default option, except for well specified and simple ToR when again the STP
could be used. The STP should require the following from consultants: a)
comments to the ToR, b) proposed work plan, d) the CVs of the proposed staff.
Suitable STP cases are implementation supervision assignments and most TA
assignments where lengthy proposal approach and methodology add little
information or value to the project.
16.4 Composition of the Shortlist
The short list should never be compromised by the inclusion of candidates with
a low level of expertise or with a dubious track record.
Applying a wide geographic spread can be a slippery exercise when done

rigidly as it may easily disregard or even contradict the consideration for
quality, leaving the Client with a suboptimal short list. According to this study,
Clients issued “mixed” or heterogeneous short lists in the sense of para. 2.8 of
the Guidelines in at least 35% of the reviewed cases, then ignored the
recommendation of the same para. 2.8 to adopt QBS or CQS in such
circumstances.
The following suggestions aim at improving the quality of the shortlist:

• Experts that prepare the ToR and lay out the selection methodology be
involved when the short list is prepared, as consultants qualifications
should match the ToR and the selection method.
• Guidelines para. 2.6 requiring six firms with wide geographic spread
should be made flexible, to prevent short listing of less qualified candidates
for the sake of geography and of number, and mitigate the risk of poor
competition.
• Para. 2.8 requiring QBS for “mixed” shortlists should be clarified
(articulated) and moved to para 3.2 as already explained under Section
16.1.8 without changing the meaning of the guidance. This would allow for
better understanding by Clients and Bank staff.
• Clients who already have a balanced and robust list of capable and trusted
consultants acceptable to the Bank that ensures sound competition, should
be exonerated from issuing the request for EoI. Clients without experience
in selecting consultants should be guided by well qualified Bank staff or
independent experts.
94
17 Composition and Operation of the Evaluation Committee

If consultants have to believe in the impartiality of the process and behave with
respect for its rules, the EC must be sufficiently qualified and honest. For the
Client the way of increasing the reliability and reducing the cost of the
selection process is not by adding criteria and checks, but to entrust it to
capable and trusted officials. The Guidelines refer to the Borrower as the party
responsible for the consultant selection without distinguishing between the EC
of the executing agency (the agent) and government (the principal). It is
recommended that the Guidelines should briefly mention the importance of i)
capacity level, composition and timely appointment of the EC, ii) procedural
rules and protocols to be followed during the technical evaluation, and iii) arm
length relationship between the Government (the principal) and the EC (the
agent) during the selection process.42
No specific provisions or text are proposed here for inclusion in the Guidelines
but the following paragraphs 17.1 to 17.3 provide a basis for preparing such a
provision if it is decided that it should exist. The Bank Consulting Services
Manual also offers detailed suggestions on this subject.
17.1 Composition of the Evaluation Committee
EC members should be appointed by the executing agency based on their

technical qualifications, familiarity with the selection process and the project,
reputation for integrity, and institutional loyalty.
In particular, it is recommended that:

• The EC should, as a rule, include the specialist(s) who prepared the ToR
and the staff months estimate as a member or at least as an
adviser/observer.
• EC members should be from the executing agency, with experience in the
disciplines of the assignment, in selecting and using consultants and
impartiality. Lacking these attributes, an independent adviser (without vote)
should be accredited to the EC by the client.
• The supervising authority clearing the evaluation report before it is
transmitted to the Bank, could attend to the evaluations as observer,
preferably without vote.
42
The Bank Consulting Services Manual provides general recommendations on both aspects
namely the characteristics of the members composing the EC and the manner in which they
should proceed prior to the evaluation, how to evaluate individually and together, and how to
prepare a conclusive technical evaluation report consistent with Bank Guideline and the RFP.
95
• The EC members should be relieved from other duties during the time the
evaluation takes place and upon appointment, EC members should be ready
to sign an integrity pact.
17.2 Operation of the Evaluation Committee
The information illustrated in Chapters 10 and 11 suggests that one should

offer some guidance and standardization, along the following lines:
• Before the opening and distribution of the technical proposals to the EC

members, the EC meets to agree on a deadlines schedule, on how to
understand the selection criteria, and how to score the proposals according
to the criteria.
• After the first review of the proposals, the EC convenes a second meeting
and each evaluator anonymously submits his scores. Anonymous scores are
recorded, and different viewpoints and information are exchanged to reduce
disagreement. If evaluators have a different assessment of a proposal they
should communicate their reasoning and insight and try to reach a
consensus on critical issues.
• After the discussion evaluators submit their scores which are then
aggregated and recorded in the evaluation report.
• The EC should be provided with simple spreadsheets that will allow
preparing orderly evaluation tables and also detect biases of various natures
that would increase the transparency of the evaluation. Examples of how
this is done are illustrated in the Annexes 1 and 2.
The anonymity of the initial “voting” is useful because people often have an
ego problem when they have openly committed to an opinion, which makes it
difficult for them to change that opinion in the face of new information.
Committee meetings are useful precisely because individual evaluators can

exchange their viewpoints and try to convince each other. In the meeting, the
test of the power of an opinion is whether it convinces others.
17.3 Adding an Independent Eye and Voice
It may be useful to add an independent adviser to the EC. This adviser should
be an expert in the field of the assignment with ample experience in consultant
selection, independent from the parties involved. He or she should have a
voice, but no vote.
96
The independent adviser will assist EC members understand each other,

respond and raise questions, express judgment and contribute to keep the
discussion on focus, mediate and help in the search for agreement.
At the end of the evaluation process, the independent adviser will sign off with
a statement either indicating that the evaluation has taken place in accordance
with best practices, or explaining why in his or her opinion this has not been
the case.
Experience suggests that an independent observer has a sobering influence, and

induces evaluators to be well reasoned, objective, and to the point. He may also
contribute to discourage or detect corruption. Altogether, his role can enhance
EC expertise and contribute to capacity building.
It is suggested that Clients use independent experts in the capacity of observers,

with voice but without vote depending from the circumstances (e.g. very
competed assignments). If this presence proves successful for purposes of
raising the quality and reliability of technical evaluations, a database of
potential experts acting as independent evaluators could be established.
18 Improving Evaluations - Weeding out Bias, Corruption,

and Shill Bidding
18.1 Improving Evaluations
It was shown under 11.1 that Clients assign close scores (average dispersion
1.12) to technical proposals that carry widely spread prices (average 1.79).
Under such circumstances price as factor of selection changes the contract
award in 36% of cases independent of ToR complexity. For assignments whose
ToR were graded “complex or very complex”, the mean technical score
increases but very marginally to 1.13 whereas the price dispersion climbs to
2.10 (see 11.1.1). Clients use QCBS in all complex cases of the sample (risk
grade 10-12) and the contract award goes to the lowest bidder in 80% of cases.
In doing so they contravene para 3.2 of the Guidelines by trading off price for
quality.
In Section 16.1 it is spelled out how the 2004 Guidelines can be rephrased so
that Clients and Bank staff interpret the guidance correctly by considering the
risk/complexity characteristics of the ToR when choosing the selection method
and the short list. It is also recommended that Bank staff advises less
experienced clients to make these considerations.
Beyond correct application of the Guidelines it is necessary, for the

procurement process to be effective and reliable that the EC members be
97
experienced and trained in appreciating technical quality as discussed in

Chapter 17. In that chapter the value of transparent collaboration between EC
members was highlighted as a tool to mitigate individual mistakes and corrupt
biases. When discussion and agreement among the EC members are difficult
to achieve, one can help it resorting to tools that identify and neutralize
different types of biases. These are described in the next section.
18.2 Weeding out Bias and Corruption
18.2.1 Use “purified” scores
It was found that evaluators frequently exhibit a “generosity bias” (Section

11.2.1). Scores that are subject to such a bias are not really comparable. In that
case, the aggregate score is of dubious quality. This suggests that one should
“purify” these scores, and extract and eliminate the generosity bias from each
evaluator’s score before one computes the average technical score.
One method for this kind of purification was explained in 11.2.1. Essentially,
this requires a transformation of the observed scores, based on the formula
illustrated in footnote 31.
This method was developed exclusively as a tool for the analysis of scoring
patterns. To actually apply purification to the computation of technical scores
for use in the actual selection process, one has to adapt the formula a bit to
bring it in a format that allows its combination with the financial score.
18.2.2 Use either “median” or “trimmed mean” aggregation
Corruption is a major concern in Bank projects. Selection rules should be

designed in such a way that corruption is made more costly and hence less
likely. Corruption frequently occurs in the form of buying influence in the EC
by bribing individual evaluators. A corrupt evaluator exercises influence by
inflating the scores that he gives to his favored proposal or by deflating the
scores given to rivals. In addition, a skilled evaluator may be able to convince
others to adopt his biased judgment.
In order to reduce the risk of corruption in the EC, one should change the
method of aggregating the scores of individual evaluators in such a way that
outliers do not affect the technical score.
Currently, the technical score is computed as the mean of all individual scores.
One could consider replacing this method of aggregation by a method that
eliminates outliers, i.e. extremely high and extremely low scores.
98
This should be done in one of two ways: either apply

• trimmed mean based, or
• median based score aggregation.
In trimmed mean based aggregation the total score is equal to the trimmed
mean of the scores of the individual evaluators. Thereby, trimming means that
one does not count the highest and the lowest x% of the scores.
Specifically, one should choose the trimming percentage x in such a way, that
the highest and the lowest scores are eliminated. For example, if there are 8
evaluators, this requires setting x = 25%; if there are 6 evaluators x = 30%. This
is precisely how the trimmed mean was computed for each project of the
sample.
An alternative to trimmed mean aggregation is “median based” score

aggregation. The median is the score of the evaluator who is “in the middle”:
one half of the evaluators rate the proposal higher, the other half rates it lower
than the median evaluator. Unlike the mean, the median is not affected by
outliers. Hence, an extreme evaluation by a single biased evaluator has no
effect on the median score.
These improved methods of score aggregation are very easy to adopt. Indeed,
the median is almost as well known as the mean, and it is, of course, readily
available in EXCEL spreadsheets. The concept of the trimmed mean is perhaps
less well known; however, it is also readily available in EXCEL spreadsheets
provided one activates the statistical functions available there.
Using one of these methods of score aggregation is a powerful and practically

costless weapon in fighting corruption.
18.3 Weeding out “shill bidding”
If the current QCBS scoring rule is employed, a consultant may benefit from
colluding with a “shill bidder” who quotes a rock-bottom price combined with
targeting a low technical score.
To detect the possibility of shill bidding one should watch for a suspiciously
low price combined with a relative low technical score and check who may
have benefited from that proposal. If one suspects shill bidding, it is, however,
difficult to deal with. Of course, one could declare the suspected shill bid
invalid. Yet, without a “smoking gun” one has no incontestable proof. Or one
could trim the extremely low ranking proposal43, provided the RFP allows it.
99
A better way to deal with shill bidding is to change the scoring rule in such a
way that shill bidding never pays. This is achieved, among other benefits, by
the new scoring rule proposed in 18.4 and detailed in Annex 6.
18.4 Improving the Scoring Rule in QCBS
In previous chapters it was shown that improper use of QCBS contradicts the
main consideration of the Guidelines and under 11.3, how the currently used
QCBS scoring formula shows undesirable characteristics. In particular, it:
• Favors the higher priced proposals;
• Unnecessarily confounds and complicates consultants’ bidding;
• Unnecessarily complicates the Clients’ calibration of the selection rule;
• Allows irrelevant proposals to change the selection, and
• Invites strategic manipulation in the form of “shill bidding”.
These flaws would have limited impact if the financial proposals showed little
dispersion. However the sample statistic shows that the dispersion of financial
proposal prices is extremely high.44 An alternative formula has been developed
in the course of this study which has the advantage of providing a constant
MRQ (Marginal Reward for Quality). This eliminates all the above listed
distortions. The formula recommended (which has been implemented in new
draft IT SBDs) avoids cases where a minor difference in technical merit points
results in an award for a price that lacks proportion to the value represented by
the point differential. The formula uses the natural logarithm of the price in
order to ensure that the desired tradeoffs between % increases in quality and %
increases in price remain constant across any price range. So, if weight of
quality is set at 0.8, the client is willing to trade 1% increase in quality for 4%
increase in price. The new formula ensures that this tradeoff remains constant
for both high and low bidders.
To illustrate the effect of the new formula consider the following scenario
applicable to the selection of a large, mission critical application assignment:
Assigned weights for price and quality are 0.2 and 0.8 respectively. Proposal 1
for $8 million garners 75 quality points. Bid 2 for $6 million garners 70
quality points. In this scenario, use of the current formula would result in
award to Bid 1, while use of the new formula would result in award to Bid 2.
43
This is sometimes done to fight shill bidding; however, it does not eliminate shill bidding, it
only raises its level of sophistication.
44
Trying to eliminate the effects of Pmin ,two solutions where tested by a) by replacing Pmin
with the second lowest price, P(min+1), or b) by replacing it with the average of all prices, Pmedium.
These arrangements do not eliminate the fundamental inconvenience of having the marginal
reward for quality depending from the offered prices and not from the Client.
100
The formula goes:
S = αSt +300 – (1 – α)ln(p)100.

where:
S= the Evaluated Bid score to be used for bid comparison,

St = the technical score, from 0.01 to 1, awarded to the bid.
α= the weight for technical merit, from 0.01 to 1.
ln(p) = the natural logarithm of the Adjusted price of the Bid;
This formula yields a measure of the spread between quality and price after (a)
reducing both to the same scale. From 0.01 to around 18, assuming that the bid
prices are lower than US$100 million and (b) applying the chosen weights.
Accordingly, the most desirable bid is that whose spread is the smallest (the
minimum S) since that bid maximizes quality and minimizes price better than
any other. The likelihood of two bids having the same score is extremely small.
If it were to happen the evaluation committee would decide.
19 Encouraging the Development of National Consultants:

Local Knowledge becomes Key
19.1 Consider the Need for Stronger Guidance
The question asked in this study is if it is enough to limit the Guidelines to

“encouraging the development of national consultants” by means of the
existing provisions listed under Chapter 12, or if the Bank Guidelines need to
better fit the consideration for quality and the Bank’s projects portfolio
increasingly focused on developing national capacities. In such case, provisions
for national consultants should be considered that could also inspire
Borrowers’ to reform their own unsuitable economic and procurement
regulations.
Voices from Clients’ such as Bangladesh, Pakistan, Colombia, Mexico, South

Africa and others indicate that promoting the participation of local firms on the
basis of price has not been a good policy as it leads to decapitalization of
consulting firms, in professional as well as economic terms.
In early Bank projects the quality of the consulting services was required for
the design of projects increasing public services delivery by adding physical
capacity, and consisting of capital goods from Part I suppliers. Increasingly, the
quality of consulting services is required for assisting projects in which local
(public sector) institutional and human capacity are the main objectives. In
these projects, the competence of nationals is as necessary to understand the
101
context and devising solutions, as the presence of international consultants to

introduce best practices. National consultants have become a necessary
ingredient, not just a (subsidiary) objective of social economic policy. The
award statistics in Section 11.1 support this fact. It is logical that the Guidelines
reflect this trend.
19.2 Short Term Recommendations
In the light of the above, without introducing changes but concentrating on a

more judicious exploitation of the opportunities offered by the existing
guidance, one could:
• Update the ceilings set in the CPARs.
• Apply CQS as the default method to consulting services assignments
limited to national consultants (including firms registered or incorporated in
the country); more explanations on the advantages offered by CQS are
provided under 16.2.
• Apply QBS or FBS to all assignments limited to national consultants that
satisfy to the criteria 3.2 to 3.4 of the Guidelines; having to negotiate
(discuss, not bargain) scope of work and contract price (QBS) improves or
clarifies the definition of the services to be delivered and the transparency
of the transaction.
• Apply QCBS and LCS for assignments where the comparison of technical
and price proposals are both justified by the nature of the assignment and
the envisaged solution; avoid comparing on price when proposed services
cannot be compared technically and on price.
The use of CQS shall shorten the duration of the selection process and reduce
transaction cost for clients and consultants. Tying the contract award to
qualifications will promote the building of professional knowledge and
reputation of domestic consulting business, as these are capital assets that
national consultants need to build. Furthermore, tying selection to
qualifications and reputation will motivate the consultant to stay on a path of
virtue.
Using QBS appropriately (for risky or complex services, heterogeneous

technical proposals) raises the consideration for the quality of services, reduces
the risk of proposing oversimplified solutions, and attracts more qualified
professionals. Qualified professionals will be attracted to a more quality
conscious public administration.
Limiting the use of QCBS prevents national consultants from the widely
practiced cutthroat competitions (e.g. Pakistan, Turkey) and from transforming
the award of professional services contracts into scoring auctions (Colombia).
102
19.3 Long Term Recommendations; Developing a Vision and a Policy
Although the participation of national consultants in Bank projects has been

expanding, albeit often in complementary roles, the odds at country level for
national consultants when procured under country rules have not turned out
favorably.
To bring about dramatic improvements to the operating environment,

procurement rules must stem from a comprehensive policy aimed at
“encouraging the development of national capacity”.
A vision at country level, where “developing national capacity” acquires an

enlarged meaning should:
i) Acknowledge that Bank lending is moving to more locally based
formulation and implementation of projects and solutions.
Correspondingly, Bank Guidelines should consider moving the presence of
national consultants from complementary to necessary. National
consultants should be acknowledged, along with international consultants,
as the carriers of professional knowledge in the proportion suiting the
project.
ii) Dispose as much as possible of the main cause of market failure in
consulting services serving the public sector, that is the asymmetry of
information between the public client and the consultants. This asymmetry
is particularly acute and stubborn when clients have little or no familiarity
with the use of consultants, especially if both are located physically and
culturally very far from each other.
iii) Avoid the illusion of fighting corrupt practices, and serving the market
better by the introduction of procurement methods that couple quality and
price competition when services are not comparable. As quality is difficult
for clients to observe or appreciate, price or bribes wrongly become
deciding factors. Replacing unsuitable procurement rules with best practice
based ones should attract the participation of national consultants and the
interest of international consultants to register as resident firms of the
country.
iv) Have national procurement regulations as compatible as possible with
standard professional practices to impede creation of parallel markets for
services. Only fragments exist today of well functioning and forward
looking national regulatory systems for consulting services that could
satisfy both requirements. The revised Guidelines could be groundbreaking
in this sense.
v) Law makers and public managers have to be bought into the benefits of the
presence of an independent consultant in a transparent democratic dialogue,
where public choices of any nature can be professionally formulated,
explained and implemented.
103
vi) On the side of consultants, it is for the consultants professional associations

to advocate integrity, high professional and technical standards among their
members and discipline them as needed, particularly when law enforcing
institutions are weak.
20 Enhancing Transparency of Evaluations

In the Guidelines transparency procedures are mainly intended to prevent open
discrimination of interested consultants and ensure some degree of
accountability by clients and their EC. It was explained (Chapter 13) that
transparency for Clients and consultants is essential but often missing where
most needed, namely in the RFP, proposal preparation and, at the evaluation
stage. Here the lack of transparency can lead directly to an inefficient contract
award. For example, technical scores differences and biases of different sorts
remain often unexplained and unresolved among evaluators and puzzling for
their appointed reviewers but can be decisive for the selection outcome
(Section 11.2).
Guidance on transparency could be articulated by:

• Concisely explaining how transparency adds value to the selection by
identifying the benefiting parties, and explaining how their mutually
transparent behavior can benefit the credibility of the evaluation process
and the commitment of the other parties.
• Indicating how a clear and unambiguous RFP promotes consultants’
willingness to compete by its rules. A statement on transparency and its
benefits would cohere with the provisions sanctioning fraud and corruption.
• Indicating the main criteria for appointment of EC members, such as
technical capacity and integrity. It is important to give consultants the
assurance that the evaluators are well chosen and unencumbered. The
possibility of considering Client appointed independent observers could
also be mentioned.
• Encouraging the use of procedural protocols outlining timing and content
of evaluation sessions. How and when to compare and discuss individual
scores, how to identify biases, reach consensus and prepare a final report.
The use of simple methods to weed out evaluators’ biases such as those
described in Chapter 18 should be considered.
• Chairpersons and evaluators become more trustworthy by the sheer
background presence of well designed procedures for complaint. This is not
because these officers will fear exposure of any abuses of power they may
be tempted to; it is because they will feel less likely to feel the temptation.
104
Eliminating the contradictions and causes for misunderstanding found in the

Guidelines as recommended under Section 16.1, would also increase
transparency.
21 Tools for Managing and Monitoring Selections

It is recommended that the data tools developed for this study be routinely
applied by Clients selecting consultants and TTL or PS who need to keep a
close eye on the selection process.
The Client would also benefit from some simulation tool to assist him in the
difficult task of calibrating the parameters of the selection process to the
particular details of the procurement case. One advantage of these tools is that
they run on Excel. Of course, the tools used in this study were not developed
for these purposes. Therefore, they would need to be adapted and improved.
EC members lacking procedure guiding the evaluation process should be

provided with protocols providing explaining:
• Understanding the ToR and assessing assignment and project risks;

• Content scheduling and management of crucial EC sessions;
• How to reach a shared understanding of the evaluation criteria and of the
rating system;
• What and how evaluators should discuss before and during the evaluation
sessions; and
• How evaluators should score proposals first individually, and then try to
reconcile their differences and reach consensus or explain their
disagreement in the evaluation report.
105
Part V
Annexes
106
V Annexes
Annex 1: Assignment’s Terms of Reference
Study on the Quality of the Guidelines on Selection and Employment of

Consultants by World Bank Borrowers: Terms of Reference of the Assignment
A. Problem Background
The policy considerations guiding of selection of consulting organizations by

World Bank Borrowers presently established in Bank guidelines are: i) quality,
efficiency and economy, ii) open and fair competition in the selection process,
iii) development and use of national consultants from developing member
countries iv) and transparency of the process of selection. The need of devising
selection rules and procedures leading to efficient contract allocation and
consistent with these considerations generates, or is hampered among others
by, the set of problems listed below.
Long and costly selection processes. The process of selection originally based
simply on consultants’ past qualifications and technical proposal quality, was
modified in different editions of the Bank “Guidelines for the Use and
Employment of Consultants” to gradually add to the list of principles indicated
above. As a result with each edition of the Guidelines (1981, 1991, 2002), the
process of selection has become increasingly time consuming and costly to
consultants and clients, because of the amount of detailed evidence to be
provided by consultants in their proposals to prove that they satisfy or comply
with the above principles. At the same time, the level of client capacity
(particularly technical) needed to evaluate ever longer and more complex
documents has not increased. Monitoring of compliance by funding agencies
is perceived by these as too burdensome, and by recipients as intrusive and
time consuming. Evidence shows that it may take easily a year to complete a
process of consultant selection.
Importance of the price factor. The introduction of price as a factor in the

formula for selection of consultants was intended to introduce price discipline
in consultant proposals, and to give a bigger chance of being selected to
consultants from poor countries with lower consultant remuneration levels. It is
not clear that this measure has achieved the proposed goals. Actually it may be
seen by good consultants as a deterrent to participation in Bank funded
107
assignments. Consultants allege that the systematic introduction of price as an

upfront factor of selection undermines innovation and efficiency of their
services and reduces the probability for good consultants of being retained by a
client because of price differences that can be negligible if compared to the
overall cost of a project. The introduction of price as a factor upfront also
sacrifices the trust required between consultant and client for a successful
execution of complex services of an intellectual nature. The excessive use of
price, independent from the type and nature of the assignment and from the risk
preferences of the client, can affect negatively the client’s interest in case of
design mistakes, when projects are complex and can have serious downstream
effects.
The problem of excessive use of price also appears to have negative

consequences when competition is restricted to national consultants from
developing countries. These are very often new firms with limited technical
and managerial experience, as well as with access to small pools of national
professional capacities.
Lagging Clients’ Capacity. When the consultant’s contract is funded by the

Bank, the evaluation of consultant proposals is carried out by the Borrower
(client). As Borrower staff often lack professional capacity, unsatisfactory
selections may take place. Corrupt practices, such as bribing of government
officers involved in the selection, and provision of false information in
proposals are also frequent. Unprofessional practices can be originated from
consultants or Borrower staff, and be driven by two different causes: on one
side the low professionalism of consultants and poorly motivated evaluators, on
the other the pressure for the consultant for being selected in spite of the
limited capacity of clients in evaluating technical proposals acceptably.
Little or no Recognition of Professional Services Quality. It is often difficult

for the Bank’s client to appreciate or measure professional quality, or
effectiveness and efficiency of consulting services in achieving the project
objectives and promised outcomes. The main reasons for such difficulties are:
a) the complexity of most public sector projects; b) the limitations of many
clients for thoroughly appreciating their projects including their technical
characteristics, objectives, and processes required to achieve those objectives;
c) the difficulty of isolating, measuring and appreciating fairly the scope and
limits of the consultant services contribution to the project quality and
outcomes. Because of these difficulties, Bank clients (and sometimes the Bank
itself) tend to concentrate their attention on process compliance, and on
economy of the inputs needed for executing the project, including the input
“consultants”, rather than paying sufficient attention to results. The evaluation
of actual project outcomes is often done often and summarily when the
consultant contract has already been concluded. The evaluation of consultant
108
performance in executing their assignment is often not done in measurable

terms of quality, effectiveness, client satisfaction, achievement of outcomes. It
generally focuses on cost, adherence to regulations, and timeliness of contract
execution within the initially established budget.
The Change in Quality Attributes. In the past few decades, the range of services
assigned to consultants in Bank funded projects has substantially increased. A
thorough updating of what are the quality attributes that are to be expected
from consultants to ensure that they contribute to the sustainability of Bank
projects has not yet been conducted nor is it reflected in the consultant
selection methodologies. The present consultant Guidelines may be more
suited for traditional and specific engineering services assignments than for
complex multidisciplinary projects requiring attention to new aspects (e.g.
sustainability and capacity building) and innovative approaches. It is also a
question if the present Guidelines can be successfully used for awarding
consulting contracts in the fields such as management and leadership,
humanitarian relief and emergency reconstruction programs.
Good Consultants Leave or Avoid the Bank funded Consulting Market of

Developing Countries. Anecdotes and interviews suggest that increasingly,
owing to all the above problems, good and reputed international consultants
from different disciplines abstain from seeking involvement in Bank funded,
Borrower executed consulting contracts. At the same time, national talented
professionals that must overcome many hurdles related to local market failures,
with lesser degrees of experience and of qualifications than international
consultants, are discouraged from engaging in a career as professional
consultants as firms as well as individuals. Both these tendencies represent
serious risks for Bank funded projects, particularly in those areas where
innovations and best professional practices are necessary to guarantee the
desired project quality. The absence of the best professional advice possible
also undermines Bank efforts of ensuring that its projects are leading examples
of modern approaches, offering innovative solutions and cutting edge
methodologies.
B. The Study Assignment
The objective of the study is to assess the effectiveness of present policies and
regulations described in the Bank’s “Guidelines for Selection and Employment
of Consultants by Bank Borrowers” in achieving efficient contract allocation
and the issues described under A. The study team would be free to propose the
overall technical approach of the assignment but the following questions would
be examined as a minimum:
109
a) Do the present Bank policies and procedures on consultant selection

and employment lead to efficient contract allocation and meet the
considerations set out in the Bank’s policies namely quality, efficiency,
economy, fair competition, national participation, and transparency? What are
the main issues affecting the present system, and is there room for
improvement and if so how? To help answer the above questions the following
should be asked:
• Do Bank Borrowers apply the most suitable selection methods

to the assignments being awarded?
• What are the most frequently adopted selection methods?
• Do the best, largest and most renowned international consultants

participate to Bank-funded competitive assignments?
• Hoe efficient i.e. how long does the process of selection take
under the present rules? Can the process be made more efficient
and flexible?
• What provisions in the existing Bank Guidelines could be added

or modified to achieve more efficient contract allocation?
• What is the level of capacity of the Borrower staff entrusted

with selection employment and management of consultants?
C. The Study Work Plan
The assessment will be based upon: a) Bank procurement statistics on the

actual use of consultants by borrowers, b) Bank project files available in the
Regions c) Survey of Bank task managers, d) Enquiries with consulting firms.
The study team will be led by the Bank Consulting Services Adviser. A senior
consultant will be engaged in the review analytically present Bank procurement
policies on consultants. Two research assistants will be assigned the collection
of data; a search is going on at present. The study is expected to require a total
of about 100m staff weeks.
Deliverables: The study team led by the Bank Consulting Services Advisor will
present the Bank a written report, commenting on present Bank policy, and
including a set of policy recommendations.
The consultant and the Bank will have a start-up meeting to define the detailed
scope of the assignment. In the course of the execution of the assignment the
110
Consultant will have access to, and will be advised as needed by Bank policy
specialists and other consultants that the Bank has identified.
The study will start in October 2005, present a preliminary report in January
2006, a final draft report in July 2006. The draft final report will be distributed
to OPCPR management, the procurement network, peer reviewers, Bank Task
Team Leaders and Procurement Specialists.
D. Funding of the Study
The study conducted by OPCPR has been supported by the contributions from
Consultants Trust Funds from Germany, Sweden, Denmark, the Netherlands
and Italy.
111
Annex 2: Sample of One Project Sheet

112
113
114
115
116
Annex 3: Purifying Technical Scores

Individual evaluators score differently, for several reasons:
• Some evaluators may be either more generous or more stingy
(generosity bias);
• Evaluators may have failed to sort out their disagreements during the
meeting of the evaluation committee, although no ill intentions are
involved, and stick to their different opinions (persistent disagreement);
• Some evaluators may be biased in favor of a particular consultant due
to corruption and use their scores to exercise influence (corruption
bias).
One useful tool to assess such differences is to look at the dispersion of
technical scores across evaluators, which is measured in Table 20 by the
standard deviation of technical scores. A high standard deviation indicates a
high level of disagreement.
In order to visualize the scoring pattern, the individual evaluators’ scores are
plotted, as illustrated in two following Figure 11 and Figure 12 (the example is
based on the real scores awarded by evaluators in one of the reviewed
assignments). There, each curve displays the scores given to a particular
proposal. If two curves do not intersect it means that evaluators rank these two
proposals in the same order. If they intersect, there is disagreement concerning
the rank order. And if a curve moves up and down it means that evaluators give
different scores to that proposal.
In the example displayed in Figure 11 it is evident that evaluators give

somewhat different scores but it seems that they tend to share the same rank
order of proposals, although this is difficult to judge since the curves almost
overlap.
Figure 11: Plot of Evaluators’ Technical Scores
Evaluators' Ranking of Consultants

120
Technical Score
100 Proposal 1
80
Proposal 3
60
40
20
0
1 2 3 4 5 6 7 8
Evaluator
117
In order to get a clearer idea of evaluators’ disagreements it is useful to

“purify” technical scores by cleaning out generosity biases. A generous
evaluator gives a high, a stingy evaluator a low average score across proposals.
This suggests that the generosity bias can be eliminated by measuring the
difference between a score and the average score of this evaluator.45
Figure 12 displays the rank order of purified scores. Since generosity biases
have been eliminated, the differences between scores displayed in this figure
reflect true disagreement, either due to honest differences of opinion or due to
corruption.
Figure 12: Plot of Evaluators’ “Purified” Technical Scores
Evaluators' Ranking of Consultants (Purified)
0.5
Technical Score
Proposal 1
Proposal 2
0
Proposal 3
1 2 3 4 5
-0.5
-1
-1.5
Evaluator
Comparing the display of the original (see Figure 11) and the purified scores
(see Figure 12), one sees that the generosity bias obscures the presence and
extent of disagreement between evaluators. Once that bias is cleaned, it
becomes clear that evaluators 1 and 2 disagree with evaluators 3, 4, and 5 over
the rank order of proposals 1 and 2. This purification is only meant to render
the preferences of each individual more visible, therefore facilitate the
comparison and reconcile scores among members, if they want.
45
More specifically, purification is done by replacing the technical score of each proposals s
∈{1, …, n} with the following transformation, S't of that score:
1 n
S ts − ∑ St
n r =1 r ,
S t′ =
σ
(where σ denotes the standard deviation of the scores given to different proposals by that
evaluator and the division by s normalizes scores by measuring them in units of standard
deviations).
118
Annex 4: A Monte-Carlo simulation of median aggregation

Table 30 illustrates the effect of median aggregation (assuming a weight of the
technical score equal to 80%). There are three proposals and seven evaluators.
The evaluators cannot measure the objective quality of the proposals precisely,
so their technical scores are not perfectly precise. We simulate this by adding
some noise when simulating the evaluators’ technical scores.46 As one can see
from the true technical scores, proposal A has the highest quality. It is also the
most expensive, but according to the scoring rule, it is this proposal that should
be chosen. Proposal B is of somewhat lower quality but is also cheaper.
According to the scoring rule, it comes in second. In the example, evaluator 1
has been bribed by the consultant of of proposal B: he inflates his technical
score for proposal B and biases his technical score against the strongest
competitor, proposal A, in an attempt to turn the decision in favor of B.
Table 30: Mean vs. median aggregation of technical scores

Proposal A B C
Price 45 40 27
Financial score 60.0 67.5 100.0
“True” technical score 60 55 45
“True” total score 60.0 57.5 56.0
Technical score evaluator 1 46 75 44
2 53 53 45
3 65 48 58
4 59 64 63
5 58 49 50
6 60 67 42
7 52 51 37
result using mean aggregation
technical score 56.1 58.1 48.4
total score 56.9 60.0 58.7
result using median aggregation
quality score 58.0 53.0 45.0
total score 58.4 55.9 56.0
The average technical scores of the three proposals, using simple averages over
all seven evaluators, are 56.1 for A, 58.1 for B, and 48.4 for C, respectively.
The total technical scores are 56.9, 60.0, and 58.7, respectively, so that
46
Specifically, we add normally distributed noise to the evaluators’ scores, plus some bias in
evaluators’ scoring, while truncating scores to [0,100].
119
proposal B is chosen. Evaluator 1’s bias has tipped the balance away from the
better choice A. If, however, the aggregate quality score would have been
evaluated using the median instead of the mean, evaluator 1 would not have
been able to make a difference, as can be seen from the table. In this case, the
aggregate technical scores are 58.0, 53.0, and 45.0, respectively, and proposal
A wins with a total score of 58.4 against 55.9 for proposal B.
The robustness of this finding was tested with a Monte-Carlo simulation.47 It

turns out that, for the particular parameters that were chosen, the aggregation of
the quality scores using simple means correctly selects proposal A with 23%
probability only. This probability increase to 47% if the median is used
instead.48 However, the flip side is that median aggregation also chooses the
least attractive proposal C with slightly higher probability, which is of course
undesirable.49
Does it make much difference?

As a routine, the two alternative methods of score aggregation are implemented
in the Excel worksheets for each sample assignment. This allows assessing
how much difference it would make if these alternative methods of score
aggregation were applied.
47
We ran 10,000 draws using R. B. Myerson’s SimTools utility, see
http://home.uchicago.edu/~rmyerson/addins.htm. The worksheet that contains the specification
of evaluators’ bias and the assumed random errors are available on request.
48
One caveat is in order here: we do not model the strategic aspects of the evaluators’
assessments. A corrupt evaluator may behave differently depending in whether the mean or the
median is used for computing the quality score. What remains true, nonetheless, is that the
median is much less affected by extreme points of view. Because corrupt evaluators are more
likely to make extreme (i.e. biased) assessments, using the median reduces the power of corrupt
evaluators, and thus decreases the bidders’ willingness to pay for corruption.
49
Mean aggregation selects project C with 4% and median aggregation with 10% probability.
120
Annex 5: Decisive bias of coalitions of evaluators

Typically, decisive influence is not exercised by an individual evaluator but by
a group of evaluators who coordinate their scoring or who share a common
assessment. If corruption is involved one should also expect that several
evaluators have been bribed. Therefore, it is important to check for
decisiveness by coalitions of evaluators.
A group of evaluators is decisive if the rank order of scores would have

changed if the scores given by the members of this group had not been counted.
A more narrow definition refers only to the highest or perhaps two or three
highest scores. In the study we looked only at decisiveness concerning the
highest ranking proposal.
Decisiveness in any form, whether exercised by one evaluator or by a group of

evaluators, indicates disagreement among evaluators. If all evaluators rank
proposals in the same order, no subgroup of evaluators is decisive because not
counting their scores would not affect the rank order obtained from aggregate
scores.
Of course, decisiveness of coalitions does not imply ill intentions. It may

simply be due to shared opinions among a few, combined with disagreement
across groups of evaluators. However, it may also be due to corruption. This
suggests that if decisiveness if identified one should perhaps take a closer look
at the nature of the disagreements between evaluators.
In the study, each of the 97 cases was analyzed for decisiveness concerning the
highest ranking proposal of coalitions. For this purpose one had to look at all
possible subset of evaluators, apply the test for decisiveness, and compute who
would have achieved the highest technical score if the scores of that group of
evaluators had not been counted.
For example, if there are 4 evaluators, {1,2,3,4}, one obtains the following
collection of 25 true subsets of evaluators:
{{1},{2},{3},{4},{5},{1,2},{1,3},{1,4},{1,5},{2,3},{2,4},{2,5},{3,4},{3,5},{
4,5},
{1,2,3},{1,2,4},{1,2,5},{1,3,4},{1,3,5},{1,4,5},{2,3,4},{2,3,5},{2,4,5},{3,4,5}
}
Therefore, in order to test for decisiveness one has to check each of the 25
coalitions of evaluators, starting with the singleton coalitions that include only
one evaluator up to coalitions of size 4.
121
If decisiveness is identified, the study records which result was prevented by

the identified decisive coalition, i.e. which consultant would have ranked
highest if the scores of that subgroup of evaluators had not been counted.
Overall, the study finds decisiveness in 34% of all cases. In other words, some
form of disagreement that upset the selection of the evaluator with the highest
technical score is found in more than one third of all evaluation committees.
This indicates significant disagreement.
Typically, one finds several coalitions that all prevent the same outcome, as
illustrated in Table 31.
Table 31: Decisive Coalitions (Contract # 48)

Decisive Evaluator
Coalition 1 2 3 4 5 6
1 (2) X x
2 (2) x X x
3 (2) x X x
4 (2) X x X
5 (2) x x X x
6 (2) x X x x
7 (2) x X x x
8 (2) x x X x x
9 (2) x x X x x
There, each row indicates a coalition, and the x’s in each row indicate that an
evaluator is a member of coalition. The numbers in parenthesis of each
coalition number indicate which other consultant would have reached the
highest technical score if the scores of that coalition of evaluators had not been
counted. Clearly, in this example, all decisive coalitions prevented that
consultant 2 reached the highest technical score. Also, some evaluators, such as
evaluators “three” and “six” figure prominently as members of each decisive
coalition.
If corruption is the driver of the observed selection, one should look at the
largest coalition that was decisive for that selection.
Of course, decisiveness is not necessarily due to corruption. However, the

Bank should check for decisiveness as a routine test, and perhaps initiate a
closer look if there are signs of suspicion.
122
Annex 6 Improving the Scoring Rule in QCBS

The mathematician can do a lot of things, but never what
you happen to want him to do just at the moment.
A friend of A. Einstein
1 Summary
The currently used scoring formula for procurement under QCBS shows
several undesirable characteristics and should be replaced.
A new formula is proposed for computing the financial score. That formula
improves the selection on several counts; in particular, it:
• eliminates the discrimination in favour of high price proposals;
• makes the rewards for quality more transparent, thus facilitating
consultants’ choice of their optimal price-quality combination;
• reduces the strategic complexity of Clients’ task of calibrating the scoring
formula to the circumstances of the procurement case;
• eliminates gains from strategic manipulation in the form of “shill
bidding”50.
Adoption of the new formula should perhaps be accompanied by raising the

stated weight of the technical score for reasons explained in Section 21.3.
Moreover, one should keep in mind that such scoring formulas are not reliable
to assess projects of high economic value, for example due to a high
downstream impact. In those cases it may be better to apply QBS.
2 The proposed new scoring formula
The currently used scoring formula for QCBS is:51
S = αSt + (1 – α)Sf (5)
pmin
Sf = 100.
p (6)
50
A “shill bidder” quotes a rock bottom price combined with a technical proposal that barely
meets the threshold level.
51
S : total score, Sf : financial score, St : technical score, α : weight of the technical score, p :
price, pmin : lowest price.
123
The proposed new formula replaces the financial score rule (6) by:
Sf = C– ln(p)100. (7)
Combining (5) and (7) gives the new scoring formula (for convenience we set
C = 300 /(1 − α ) ):
S = αSt + (1-α) C – (1 – α)ln(p)100
= αSt +300 – (1 – α)ln(p)100. (8)
Remark 2 (Technical) The formula has two degrees of freedom. Every affine
transformation of S, i.e. every S′= γS + c with γ > 0 and c = const. is equivalent
in the sense that it yields the same selection. Because if a proposal ( Sti , pi)
reaches the maximum score for a given choice of parameters (γ,c), then it
evidently also achieves the maximum score for all other choices of these
parameters.
We chose C = 300 /(1 − α ) only because it generates “nice” numbers in our

examples.
Note: the Guidelines (The World Bank (2004b)) do not prescribe a particular
rule for computing the financial score. Therefore, if the new rule shall be
applied, one does not need to change the Guidelines; one only needs to change
the Manual (The World Bank (2006)).
The section closes with Table 32 that illustrates the selection based on the old
and the new formula, assuming α = 0.8. In this example, the old formula
selects proposal 4 (with a technical score equal to 98) and the new formula
selects proposal 2 (with a technical score equal to 92). The selection based on
the new formula marked by light shading; that based on the old formula by
dark shading.52
52
This example is designed to illustrate; it is not claimed that it is representative.
124
Table 32: Selection by old and new scoring formula assuming a = 0.8
Proposal Technical Price Total Score S Total Score S
Score St p New Formula Old Formula
1 65 1,938,436 62.45 54.07
2 92 592,779 107.75 80.38
3 40 592,779 87.78 52.00
4 98 844,914 105.46 83.16
5 83 2,715,094 70.11 67.88
6 58 1,150,855 67.28 49.89
3 Why this formula?
Since technical proposals are scored relative to best practice (the latter is
assigned the score 100), the technical score St is a percentage point, such as St =
80% or St = 90%. Therefore, every change in St, denoted by dSt , is a relative
change, expressed in percentage points. This dictates that price must also be
scored in such a way that a change in the price score reflects a relative price
change, expressed in percentage points.
How can one measure the marginal reward for quality that is implicit in a given
scoring formula? As already explained in Section 11.3.9, the right measure of
the effective weight of the technical score is the Marginal Reward for Quality,
MRQ, between quality and price.
Loosely speaking, that MRQ says by how many percentage points a consultant
can raise his price, dp/p in exchange for a 1% increase in technical score, dSt =
1, without changing the total score. Its exact definition is as follows:
dp/ p
MRQ := 100
dSt S = const . (9)
As explained in Section 11.3.9, the MRQ of the currently used scoring formula
is increasing in price and decreasing in the smallest quoted price. Hence it
favors high priced proposals, especially if the dispersion of prices is high.
Moreover, this rule makes it exceedingly difficult for bidders to assess that
reward for quality, because it depends upon the behavior of other bidders. This
unnecessarily complicates the complexity of bidding.
In view of the various deficiencies of the currently used formula, the new
formula (8) was designed in such a way that it keeps the marginal reward
for quality independent of the own price and of the prices of other consultants.
125
In order to show that the new scoring formula has this property, totally
differentiate (8) and set dS = 0, and one finds (using the fact that
dp
dp ln( p ) = p ) :
d
dp / p α
100 = .
dS t 1−α
MRQ-new := S = const. (10)
Therefore, under the new formula the marginal reward for quality is equal to
α/(1 – α), which is independent of prices.
That ratio α/(1 – α) has a straightforward interpretation: if a consultant raises

his technical score by dSt = 1%-points, his total score goes up by α%-points; if
he raises the price by 1%-point, the score falls by (1 – α)%-points. Altogether,
for each 1%-points increase in technical score he can raise his price by α/(1 –
α)%-points without changing his total score.
The new formula exhibits a lower MRS than the old, except for the bidder who
quotes the lowest price, p = pmin (see also Figure 13):
α α p min
>
p > pmin ⇒ MRQ-new = 1 − α 1 − α p = MRQ-old. (11)
Therefore, the adoption of the new formula should be accompanied by raising

α, say from α = 80% to α = 85%.
MRS
Standard
New
α
1−α
p
pmin
Figure 13: Relationship between the MRS and price

126
Another reason for raising α is the following. Clients sometimes recognize that
the currently employed scoring formula may have a built-in bias in favor of
proposals that exhibit a high price combined with a high quality. In order to
mitigate this bias, the stated weight of quality, α, is sometimes artificially
lowered, say from α' = 85% to α'' = 80%.
Under the new formula there is no built-in bias that needs to be mitigated.
Therefore, if the new formula is adopted one should undo the artificial
lowering of α and, for example, replace α = 80% by α = 85%. It is of crucial
importance to keep these facts in mind.
Of course, acceptance of the new formula requires that one feels comfortable
with computing the natural log of prices, ln( p ) . This may be a bit unfamiliar, at
first glance. However, it should not pose a practical problem. The ln function is
one of the oldest functions that has been widely used to simplify computations
for centuries, assisted by large scale availability of log-tables.53 Today, the old
log-tables have become redundant since the log function is readily accessible in
pocket calculators and in spread-sheet software such as MS Excel. There is no
excuse for not using it.
4 Another comparison of the “old” and “new” scoring rule
In Section 11.3.9 we showed how, under the “old” formula, increasing prices
requires a compensating increase in the technical score at a diminishing rate. It
may be useful to briefly return to these examples and show how the new rule
compares to the old in these examples.
Table 33 lists all proposals that score the same, in the order of increasing
prices, for both the “old” and the “new” scoring rule. And Figure 14 plots these
score-equivalent proposals for a finer grid of prices. As one can see, the old
formula consistently favors high prices (the curve for the “old” formula is
consistently below that of the “new”). Under the old formula, one always finds
a technical score below 100 that compensates for a higher price, no matter how
high that price is. Whereas, under the new formula, one cannot raise the price
above an amount close to 3,000 and compensate a high price by a higher
technical score (which is bounded from above by 100).
53
In fact, the first logarithmic tables were produced by Muslim mathematicians as early as in
the 13th century.
127
Table 33: (St, p) combinations that score the same

Proposal St St P
Old Rule New Rule Price
1 70.00 70.00 $1,000
2 72.27 72.38 $1,100
3 82.50 87.33 $2,000
4 86.67 97.47 $3,000
5 88.75 104.66 $4,000
6 90.00 110.24 $5,000
Figure 14: (St, p) combinations that score the same
In our sample we have altogether 86 cases where QCBS was employed,

combined with various weights of quality, α . We find that if the new formula
had been used, the selection would have changed in 3.5% of all cases if the
switch had not been accompanied by a change of α . However, if the switch
were accompanied by an increase of α in the order of 6% (so that α = 0.8 is
replaced by α = 0.85 ), which is generally advised for reasons explained above,
adopting the new formula would have changed the selection in 8.14% of all
cases, which is a considerable effect.
5 Generalization to “diminishing returns to quality”?
Why should the MRQ be constant?
Note, we required that the MRQ should be independent of prices, and thus
constant in the own price and the prices of other consultants. However, this
does not mean that it should also be independent of quality, measured by the
technical score.
128
Indeed, the MRQ should be constant in prices, but not necessarily constant in
the technical score, S t.
If there are diminishing returns to quality, because further and further increases
in quality become less valuable, one should adjust the formula in such a way
that the MRQ becomes monotone decreasing in St.
For example, if one procures an automobile, one is willing to pay more for
higher quality, but that willingness to pay diminishes as quality is increased. If
this were not the case, we would all drive only a top of the line automobile like
a Rolls-Royce.
If one wants to employ a scoring rule that reflects some diminishing returns of
quality, one can easily generalize the proposed formula (12) as follows:
S = αStγ + (1 – α)(C-ln(p)100), with 0 < γ < 1. (12)
(Similarly, if for some reason there are increasing returns to quality, one should
adopt that generalized scoring formula and set γ > 1.)
If diminishing returns to quality set in only above a certain threshold level, say
above St =90%, one can switch to the above for all St ≥ 0. 54
6 Calibration: “No one–size–fits–all”
Regardless of which scoring formula is employed, the client needs to calibrate

the parameters α,Stmin, to the specific characteristics of the procurement case.
No one-size fits–all.
From what was observed in the sample of contracts, clients do not seem to pay
much attention to calibration. They tend to choose the two parameters
independently, and they tend to choose the same numbers across the board.
This suggests that the WB should develop a methodology and provide technical
tools for calibration to assists clients. For example, one could develop
simulation tools combined with building a stochastic model of price quality
choices based on “resampling” past observations from similar procurement
cases.55 Such tools should be an essential part of a methodology for calibration.
54
One may also ask, why not use a more symmetric scoring format such S = γ ln(St) – δ
ln(p)100, or equivalently the familiar Cobb–Douglas functional form. However, notice that St
is already a relative whereas p is an absolute number. This should explain why the seeming
asymmetry is warranted.
55
Resampling is a useful approach to model uncertainty based on past observations. It is the
basis of the methodology of “bootstrapping”.
129
7 Beware of the limits of relative scoring
As already pointed out in Section 11.3.2, scoring rules that employ a relative
measure of the quality of the technical proposal have serious limitations. If
projects have a high economic value, these rules cannot be relied upon to make
a good selection.
Ideally, one should score proposals by their value-added, and select the project
that promises the highest value added (see the discussion in Section 11.3.2).
Unfortunately, in many practical applications it is intrinsically difficult to
estimate that value added or the Client or his evaluation committee lack the
expertise to compute it. This is why substitute scoring rules like the one
currently used or the one proposed are employed.
These substitute rules cannot assure a reasonably good selection if the absolute
value of the procurement project is high, for example because there is
considerable downstream impact. In these cases one should not use these rules
and, to be on the safe side, replace QCBS by QBS, followed by price
negotiations with the technically most suitable consultant.
Milestone Changes in Bank Guidance on Consultants
Number Quality and Cost as

Edition Considerations Evaluation Criteria Shortlist
of Pages Selection Factors
Annex 7:
09/1966 8 “A basic requirement, in “Proposals (…) should be carefully “It should be clearly indicated that “The selection (…) should
addition to being fully analyzed and compared with respect to financial terms are not desired at this begin with the preparation of a
competent for the task at • plans of approach, stage; that selection will be made reasonably sized list of firms
hand, is that the selected firm • schedules, entirely on the basis of claiming expertise in the field.
shall be from a member • experience and capabilities of qualifications to perform the work The list may then be shortened
country or from Switzerland” personnel to be assigned, and not on price” (see para. 4.1) by detailed studies of each
(see para. 1.1) • the quality of supervisory firm’s experience and
leadership, capabilities until four or five
• attention to be given by principals remain” (see para. 4.1)
of the firm,
• facilities of the home office, and
• the assistance, if any, that may be
available from others” (see para.
4.2)
04/1974 19 “In evaluating proposals (…), “In evaluating proposals (…), “The Bank normally invites
borrowers should focus primarily on borrowers should focus primarily on three to five firms to submit
• each firm’s current professional each firm’s current professional proposals for one study
qualifications, qualifications, its recent assignment. (see para 1.32).
• its recent performance on similar performance on similar assignments, “The Bank suggests that no
assignments, its understanding of the particular more than five firms be
• its understanding of the particular assignment at hand, and the staff” invited; it also encourages
assignment at hand, and (see para. 1.24). Except in cases of borrowers to make the list
• the staff it has available when the very simple and well-defined internationally representative
work is required” assignments for routine projects, with, say, not more than two
(see para. 1.24). the Bank recommends that the firms from one country” (see
borrowers invite proposals without para. 1.22)
financial terms, that they select a
firm solely on the basis of
Historical Development of Bank Guidelines
qualitative considerations
mentioned above, and financial
terms be agreed later, during
contract negotiations” (see para.
1.26)
130
08/1981 38 “It is the Bank’s policy to “For certain types of assignment, it may The two basic types of selection “The Bank (…) strongly
encourage and to foster the be appropriate to take into account, in procedures are, first, those that rely recommends a short list of
development of domestic addition to the technical evaluation, the solely on an evaluation of the firms with a lower limit of
consulting firms. The Bank cost to the Borrower of the services technical competence, the three and an upper limit of
encourages borrowers to employ offered. The process of selection, with personnel undertaking the six” (see para 2.14)
domestic consulting firms (…) or without price, should maintain assignment, and the suitability of its
either alone or in combination quality as the paramount requirement proposal; and, second, those that
with foreign firms” (see para. of consulting services. Price must not involve both a technical evaluation
1.11). dominate the selection process to the and consideration of the offered
detriment of the effective execution of price of the services” (see para.
the project. Nevertheless, by inviting 2.23)
consulting firms to submit priced
proposals for certain types of assignment
for which price comparisons can Explicit mention of SSS (see paras.
approximately be a factor, borrowers may 2.02, 2.16-2.20)
be able to take advantage of cost savings”
(see para 2.28 – Technical Evaluation
with Price Consideration)
01/1997 52 “Five main considerations guide “The evaluation of the proposals shall be “The Bank considers that, in the “The Borrower is responsible
09/1997 the Bank’s policy on the selection carried out in two stages: first the quality, majority of cases, these for the preparation of the short
01/1999 process: and then the cost” (see para 2.13- considerations can best be addressed list. (…) Short lists shall
05/2002 • the need for high quality Evaluation of Proposals: Consideration of through competition among comprise three to six firms
services Quality and Cost) qualified short-listed firms in which with a wide geographic
• the need for economy and the selection is based both on the spread, with no more than
efficiency quality of the proposal and on the two firms from any one
• the need to give all qualified cost of the services provided country and at least one firm
consultants an opportunity to (Quality and Cost-Based Selection from a developing country”
compete – QCBS) (…) However, there are (see para 2.6)
• the Bank’s interest in cases when QCBS is not the most
encouraging the use of appropriate method of selection. For “The short list may comprise
national consultants in its complex or highly specialized entirely national consultants
developing member countries assignments or those that invite (…) if the assignment is below
• the need for transparency in innovations, selection based on the the ceiling established in the
the selection process” quality of the proposal alone Loan Agreement” (see para
(see para 1.4) (Quality-Based Selection – QBS) 2.7)
(…) would be more appropriate. (see
para 1.5)
131
05/2004 58 “Five main considerations guide “The evaluation of the proposals shall be “The Bank considers that, in the “The Borrower is responsible
the Bank’s policy on the selection carried out in two stages: first the quality, majority of cases, these for the preparation of the short
process: and then the cost” (see para 2.14) considerations can best be addressed list. (…) Short lists shall
• the need for high quality (…) through competition among comprise six firms with a
services “The borrower shall evaluate each qualified short-listed firms in which wide geographic spread, with
• the need for economy and technical proposal (using an evaluation the selection is based on the no more than two firms from
efficiency committee of three or more specialists in quality of the proposal and, where any one country and at least
• the need to give all qualified the sector), taking into account several appropriate, on the cost of the one firm from a developing
consultants an opportunity to criteria: services provided. country” (see para 2.6)
compete • the consultant’s relevant (…)
• the Bank’s interest in experience for the assignment Quality and Cost-Based Selection “The short list may comprise
encouraging the use of • the quality of the methodology (QCBS) is the most commonly entirely national consultants
national consultants in its proposed recommended method” (see para (…) if the assignment is below
developing member countries • the qualifications of the key staff 1.5) the ceiling established in the
• the need for transparency in proposed Procurement Plan approved
the selection process” • transfer of knowledge, if required in by the Bank” (see para 2.7)
(see para 1.4) the TOR
• the extent of participation by
nationals among key staff in the
performance of the assignment
Each criterion shall be marked on a scale

of 1 to 100. Then the marks shall be
weighted to become scores” (see para
2.15)
132
133
Annex 8: Selection Rules of Other Agencies

A Comparison of the World Bank Guidelines for the Selection & Employment of
Consultants with Procurement Guidance of Selected Bilateral Donors, MDB’s and Other
Relevant Organizations
Regulation Selection Number of Responsible Composition of
Methods Firms on for Evaluation Committee
Short List Procurement
World Bank Guidelines QCBS (with 20% 6 firms Borrower To be determined by
weight given to Borrower
price), QBS, LCS,
FBS, CQS, SSS
AusAID Commonwealth QCBS (with 10- 3-5 firms AusAID EC includes 2 voting
Procurement 20% weight given AusAID members, 1
Guidelines & to price) voting independent
AusAID member and non-voting
Procurement AusAID members
Policy
Framework
Cida Canadian QCBS (with 20% Applies for Executing EC shall include
Treasury Board weight given to assignments Agency minimum of 3 members,
Contracting price) $100.000 or (could be normally all from CIDA.
Policy, smaller. either One member is the
Government Minimum 3 Canadian or project manager (head
Contracts firms with from evaluator) and the others
Regulations and no upward Recipient are sector specialists. An
the CIDA limit Country) independent consultant
Procurement may be included as voting
Handbook for member on the EC for
Goods and more complex
Related Services infrastructure
assignments. A
representative from the
Client may also be
included as voting
member on the EC.
Danida EU regulation QCBS (with 20- N.A. but Danida or EC should comprise at
and Manual 30% weight given long list Recipient least 5 members:
to price) comprise Country 1 chairperson from
5-8 firms Danida’s
Business Cooperation &
TA
1 member from
responsible Danish
Diplomatic Mission
1 member from Danida’s
TA Service
1 observer from the
Recipient Country
134
1 independent
consultant
DFID EU regulation QCBS (with 20% 6 firms in DFID EC should comprise at
and Guidelines weight given to average least 4 DFID members.
(The DFID Blue price), SSS The following
Book – professionals should be
Essential Guide included:
to Rules and Project officer
Tools) Professional Adviser (s)
Partner (s)
Contract officer
Independent expert from
private sector may be
included if DFID
members lack the
required capacity
Finnida EU regulation QCBS (with 20% 3-6 firms Finnida N.A.
and Guidelines weight given to N.B. Finnida
price), SSS carries out an
investigation
whenever
there is
significant
disparity
between
prices offered
for a service,
in particular
when public
bodies or
NGO’s are
competing
alongside
private
companies
GTZ & EU regulation QCBS (with 20- Maximum 6 GTZ (GTZ as For GTZ contracts of
KfW and Guidelines 30% weight given firms (GTZ) executing Euro 200,000 and above,
(GTZ and KfW) to price) GTZ and Maximum 5 agency) the EC comprise 4 GTZ
KfW firms (KfW) Recipient members:
SSS Country 2 sector specialists
(KfW as 1 senior project officer
financing 1 contract officer
institution) Final ranking of the
technical proposals are
done before opening the
financial proposals. An
independent consultant
may be included and act
on behalf of a sector
specialist if the required
technical expertise is not
available in-house
(uncommon and
135
discouraged as GTZ finds

it difficult to ensure the
impartiality of such
members).
JBIC & Guidelines QBS 3-5 firms Recipient Number of EC members
JICA (JBIC) (JBIC) Country unknown but an
Up to 15 (JBIC) independent member
(JICA) JICA may be included. Specific
mentioning in Guidelines
that EC should be
established before the
preparation of the ToR
(JBIC)
All EC members are JICA

staff (JICA)
Sida EU regulation, QCBS (with 20- 3-6 firms Recipient 3-4 voting members from
Guidelines and 40% weight given Country or Sida. An independent
Manual to price), LCS, Sida consultant and a member
SSS, Framework from the Recipient
Country may also be
included on the EC
USAID Federal QBS for 3-6 firms for USAID, or For general consulting
Acquisition Engineering/ large (very rarely) contracts the evaluation of
Regulation – Architect contracts contracts Recipient technical and financial
FAR (paras (10% of USAID 2-3 firms for Country proposals are done
15,16,36 & 37) contracts) small simultaneously but
and USAID contracts independently by different
Acquisition QCBS (with 20- panels.
Policy (ADS 40% weight given
300) to price) for most For engineering
other consulting consulting contracts the
services contracts technical evaluation is
(90% of USAID done entirely based on the
contracts) firms quality. Interviews
are then carried out with
LCS, although the three highest ranking
very rarely firms.
EC panel comprise 3-5
USAID staff.
Independent members
may be included from the
private sector
136
FIDIC Guidelines QBS preferred, 3-6 firms Client FIDIC recommends

QCBS (with 1- (normally balanced inter-agency EC.
20% weight given 3-4 firms for 1 independent
to price), FBS, complex consultant or consulting
LCS contracts & firm may be included on
SSS 4-5 firms for EC or apply the use of a
simple combined public-private
contracts) panel
56 QCBS (with 20% 4-8 firms Borrower The EC shall comprise 1
European Guidelines
Commission weight given to non-voting chairperson, 1
(Budget and price), Framework non-voting secretary and
EDF – an odd number (minimum
European of 3) of voting technically
Development and administratively
Fund) capable members. A non-
voting independent
observer may be
appointed by the EC
57 QCBS (with 20% 6 firms ADB EC composed of ADB
ADB Guidelines
weight given to (However a project staff and sector
price), QBS, LCS, pilot project specialists
FBS, CQS, SSS, with 15
Framework assignments ADB applies CRAM
is underway (Consultant Recruitment
where Activity Monitoring) to
responsibility monitor consultant
for Technical selection process duration
Evaluations
lies with the
Borrower)
56
“The procurement rules that apply to all European Community external aid contracts
financed from the European Communities general budget and the European Development Fund
are very similar to the ones adopted by the World Bank. The European Development Fund
employs the same two envelope system and they also use the same flawed scoring rule for the
financial proposal. The tender with the lowest total fees receives 100 points. The others are
awarded points by means of the following formula: Financial score = (lowest total fees/total
fees of the tender being considered) × 100 (The European Union, 2006, p. 46). The only
significant difference is that they generally use a price cap which is referred to as the “budget”.
The total contract value comprises the fees (including employment-related overheads), the
incidental expenditure and the provision for expenditure verification, which are specified in the
tender dossier. This total contract value is compared with the maximum budget available for
the contract. Tenders exceeding the maximum budget allocated for the contract are eliminated”
(The European Union, 2006, p. 46).
57
“The procurement rules of the Asian Development Bank are even more similar to those
employed by the WB. Their financial score rule differs only by multiplying all number by
1,000 instead of by 100” (The Asian Development Bank, 2002, p. 34ff.).
137
Annex 9: Example of Project Procurement Plan
PARAGUAY: Road Maintenance Project - Procurement Plan

1 2 3 4 5 6 7
Ref. Description of Estimated Selection Review by Expected Comments

No. Assignment Cost Method Bank Proposals
(Prior/ Submission
Post) Date
1 Supervision of 700,000 QCBS Prior nn Shortlist
Performance- Through
Based Expressions of
Maintenance Interest
Contract 1
Contract 5
Contract 0
Contract 2
Road Through
Improvement Expressions of
Works in Interest
Caaguazu Depart.
(3 contracts)
Road Through
Works in San Interest
Pedro Depart (4
contracts)
Road Through
Works in Interest
Caazanpa Depart
138
(3 contracts)
8 Preparation of 2nd 360,000 QCBS Prior nn Shortlist

and 3rd year Through
Works in Expressions of
Unpaved Road Interest
Network
9 Indigenous 300,000 QCBS Prior nn Shortlist
People’s Plan Through
Implementation Expressions of
Interest
10 Update of the 1,450,000 QCBS Prior nn Shortlist
SIAMV and the Through
Road Inventory Expressions of
Interest
11 Socioeconomic 150,000 QCBS Prior nn Shortlist
Impact Survey - Through
Baseline Expressions of
Interest
12 Strengthening and 100,000 QCBS Prior nn Shortlist
Training of UPGP Through
Expressions of
Interest
Training of Through
Government and Expressions of
Municipalities Interest
Staff
Improvement of Through
the Toll Expressions of
Collection Interest
System
15 Design of a 100,000 QCBS Prior nn Shortlist
Communication Through
Strategy of Road Expressions of
139
Bibliography
AKERLOF, G. (1970): “The market for lemons, qualitative uncertainty, and the market
mechanism,” Quarterly Journal of Economics, 84, 488–500.
ANDERSON, J., AND M. FROHLICH (2001): “FreeMarkets and Online Auctions,” Business
Strategy Review, 12, 59–68.
ARVIS, J.-F., AND R. BERENBEIM (2003): Figthing Corruption in East Asia: Solutions
from the Private Sector. The World Bank, Washington, D.C.
ASKER, J., AND E. CANTILLON (2004): “Properties of Scoring Auctions,” Working paper,
Harvard Business School.
BAJARI, P., R. MCMILLAN, AND S. TADELIS (2002): “Auctions vs. Negotiations in
Procurement: An Empirical Analysis,” Working paper, Department of Economics,
Stanford University.
BRAMS, S. J., AND P. C. FISHBURN (1983): Approval Voting. Boston.
BRYAN, M. F., AND S. G. CECCHETTI (1994): “Measuring Core Inflation,” in Monetary
Policy, ed. by N. G. Mankiw, pp. 195–215. University of Chicago Press, Chicago.
CHE, Y.-K. (1993): “Design Competition through Multidimensional Auctions,” Rand
Journal of Economics, 4, 669–680.
JACOBS, J. (2004): Dark Age Ahead. Vintage Books.
JAP, S. (2002): “Online Reverse Auctions: Issues, Themes and Prospects for the Future,”
Journal of the Academy of Marketing Science, 30, 506–525.
KAUFMANN, D. (2005): “Six Questions on the Cost of Corruption with World Bank
Institute Global Governance Director Daniel Kaufmann,” in News-The World Bank,
2005. Washington, D.C.
LAFFONT, J.-J., AND J. TIROLE (1991): “Auction Design and Favoritism,” International
Journal of Industrial Organization, 9, 9–42.
LEE, T. A. (1999): “The Experience of Singapore in Combating Corruption,” in Curbing
Corruption. Toward a Model for Building National Integrity, ed. by R. Stapenhurst,
and S. Kpundeh, pp. 59–66. Economic Development Institute of The World Bank,
Washington, D.C.
LENGWILER, Y., AND E. WOLFSTETTER (2005): “Bid Rigging. An Analysis of Corruption
in Auctions,” Discussion paper, Humboldt-University at Berlin.
⎯⎯ (2006): “Corruption in Procurement Auctions,” in Handbook of Procurement —
Theory and Practice for Managers, ed. by N. Dimitri, G. Piga, and G. Spagnolo.
Cambridge University Press, forthcoming.
MILGROM, P., AND R. J. WEBER (1982): “A theory of auctions and competitive bidding,”
Econometrica, 50, 1089–1122.
140
SMITH, A. (1776): The Wealth of Nations. Cannan Edition, Modern Library, New York,
1937.
STAPENHURST, R., AND S. J. KPUNDEH (eds.) (1999): Curbing Corruption. Toward a
Model for Building National Integrity. EDI Development Studies, The World Bank,
Washington, D.C.
STATE OF ARIZONA (1999): “Arizona Department of Transportation A+B Bidding,
Performance Audit,” Discussion Paper Report No. 99-22, Office of the Auditor
General.
THE ASIAN DEVELOPMENT BANK (2002): Handbook for Users of Consulting Services:
Procedures and Practices. Asian Development Bank, Manila.
THE ECONOMIST (2004): “Just how rotten?,” The Economist, pp. Oct. 21, 2004.
THE EUROPEAN UNION (2006): Practical Guide to Contract Procedures for EC External
Actions. http://europa.eu.int/comm/
europeaid/gestion/index_en.htm.
THE WORLD BANK (2004a): Guidelines: Procurement of Goods and Services by World
Bank Borrowers. The International Bank for Reconstruction and Development. The
World Bank, Washington, D.C.
⎯⎯ (2004b): Guidelines: Selection and Employment of Consultants by World Bank
Borrowers. The International Bank for Reconstruction and Development. The World
Bank, Washington, D.C.
⎯⎯ (2006): Consulting Services Manual. A Comprehensive Guide to the Selection of
Consultants. The International Bank for Reconstruction and Development. The World
Bank, Washington, D.C.
TREPTE, P. (2004): Regulating Procurement. Oxford University Press.
WEBER, R. J. (1995): “Approval Voting,” Journal of Economic Perspectives, 9, 39–49.
WOLFSTETTER, E. (1996): “Auctions: an introduction,” Journal of Economic Surveys, 10,
367–421.
⎯⎯ (2003): Topics in Microeconomics: Industrial Organization, Auctions, and
Incentives. Cambridge University Press., 2nd edn
wb102197
C:\Documents and Settings\wb102197\My Documents\Study on Consulting Serivces Quality\Study Final Version\GLS Draft Final
Version January 27,2007.doc
01/29/2007 6:43:00 PM

World Bank Policy on Selection and Employment of Consultants

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

World Bank Policy on Selection and Employment of Consultants

Transféré par

Droits d'auteur :

Formats disponibles

WORLD BANK POLICY ON SELECTION AND

EMPLOYMENT OF CONSULTANTS: STUDY OF ITS

Giovanni Casartelli Elmar Wolfstetter

THE WORLD BANK

Accronyms and Abbreviations....................................................................... 10

II Study Purpose and Design ............................................................22

5 The Study’s Objective, Approach and Target Audience .................... 23

6 The Study’s Design ................................................................................. 24

6.1 Characteristics of selected assignments ......................................... 24

III Findings ..........................................................................................30

7 Efficiency and Economy ......................................................................... 30

9.1 Terms of Reference ........................................................................ 36

10 The Evaluation Committee .................................................................... 51

11 The Second Phase of the Selection Process: Proposal Evaluation...... 54

11.1 The Evaluation of Technical Proposals ......................................... 54

12 National Consultants Participation ....................................................... 75

12.1 Part II Consultants participation rates ............................................ 76

13.1 Clarity and Comprehensiveness of Guidelines and RFP ............... 80

14 Warning Signs ......................................................................................... 81

15 Raise Efficiency and Economy............................................................... 85

15.1 Bundel Assignments and Allow Continuations ............................. 85

16 Raise Consideration for Quality ............................................................ 87

16.1 Introducing the Consideration for Risk.......................................... 87

17 Composition and Operation of the Evaluation Committee................. 94

17.1 Composition of the Evaluation Committee.................................... 94

18 Improving Evaluations - Weeding out Bias, Corruption, and Shill

18.1 Improving Evaluations ................................................................... 96

19 Encouraging the Development of National Consultants: Local

19.1 Consider the Need for Stronger Guidance ................................... 100

20 Enhancing Transparency of Evaluations............................................ 103

21 Tools for Managing and Monitoring Selections................................. 104

Annex 1: Assignment’s Terms of Reference ........................................... 106

Annex 2: Sample of One Project Sheet.................................................... 111

Annex 4: A Monte-Carlo simulation of median aggregation ................ 118

Annex 5: Decisive bias of coalitions of evaluators .................................. 120

Annex 6 Improving the Scoring Rule in QCBS...................................... 122

1 Summary ...................................................................................... 122

Annex 7: Historical Development of Bank Guidelines........................... 130

Annex 8: Selection Rules of Other Agencies ........................................... 133

Annex 9: Example of Project Procurement Plan ................................... 137

Bibliography .................................................................................................. 139

Accronyms and Abbreviations

i) The Guidelines considerations are not being met—particularly with

3. In light of these findings, this report recommends amendments designed

4. It is important to add that, while changing the Guidelines will help

6. Quality. The complexity and specialization of consultant services keep

7. Borrowers’ competence in preparing Terms of Reference (ToR), in

11. Fair Competition. Borrowers generally comply with the requirements

12. Development of National Consultancies. Bank statistics and the study

13. Transparency. The consideration of transparency was patched into the

14. Conclusion. The outstanding conclusion of the study is that rigid or

16. Key Recommendation 1: Reduce complexity. This can be achieved

18. Key Recommendation 3: Stress the importance of the “qualifications

19. Key Recommendation 4: Affirm the importance of a professionally

Encourage Development of National Consultants

21. Key Recommendation 6: Promote development of the national

22. Key Recommendation 7: Improve Guidelines treatment of

23. As the complexity of projects increases, the specialization and

Study Purpose and Design

II Study Purpose and Design

Although consultants’ services have evolved to accommodate changing

Specific concerns from Bank clients, Bank operational staff, executive