1660 Hollis Street,

5th Floor Centennial Bldg

Halifax, Nova Scotia B3J 1V7

Tuesday, September 26, 2017

Mr. Alan Haaksma,

Director of Business Development
Evident Canada
alan.haaksma@evident.com

Re: RFP # WS53233446 – OPOR Clinical Information System – Complaint

Good day Mr. Haaksma,

This communication is in response to your letter received in August, 2017 regarding RFSQ WS53233446 -
One Person-One Record Clinical Information System, submitted in accordance with the Province of Nova
Scotia’s Procurement Complaint Protocol.

In accordance with the Protocol, a Procurement Review Committee (“PRC”) was established by Internal
Services, Procurement Branch (“Procurement”) in order to review the details of the complaint. The
scope of the review consisted of the procurement aspects of the complaint, covering the following
areas:

i. Clarification process followed during the evaluation phase;

ii. Page length for responses to the solicitation;
iii. Scoring system and evaluation criteria;
iv. Conflict of interest; and
v. Any other aspect of the complaint deemed relevant by the PRC.

The PRC has completed their review and has concluded the following:

1) Clarification process followed during the evaluation phase

 The PRC confirmed that the clarification process followed during the RFSQ evaluation was in line
with proper procurement standards and the process identified in the RFSQ.
 The only clarifications sought were made during the rectification period (as stated in Section
2.1.1. of the RFSQ). No further clarifications were requested of any Respondent during the
evaluation. Also, as stated in the RFSQ, the rectification period was only for mandatory
submission requirements.
 Furthermore, pursuant to the Canadian International Trade Tribunal (CITT), a clarification is
acceptable only if it is an explanation of some existing aspect of the proposal that does not
amount to a substantive revision or modification of the proposal. Respondents cannot be
allowed to repair their proposal through a clarification process.
 1660 Hollis Street,
5th Floor Centennial Bldg
Halifax, Nova Scotia B3J 1V7

 Clarifications, during the procurement process, would not have been an opportunity for
Respondents to provide additional information but simply to provide an understanding of the
response provided.
 Proposals are evaluated on the content provided therein; outside information is not traditionally
gathered or obtained through other sources (i.e., internet). The onus is on the Respondent to
ensure their proposal is complete. Though the Respondent noted some of the information was
readily available on the internet, the Province would not be in a position to evaluate such
content.

2) Page length for responses to the solicitation

 The PRC confirmed that the process identified in the RFSQ was followed by the evaluation
committee, that is, the page lengths were suggestions only and were not used as part of the
evaluation process, therefore scores were not affected. Respondents also had the right to
exceed such page lengths should they have required it.
 As defined in the RFSQ, Section D. Rated Criteria, Suggested Response Length:
Suggested response length (pages) for some questions are provided to guide
Respondents in answering in a clear and concise manner that addresses the questions.
The suggested length is not a limit and should they require it, Respondents are
permitted to elaborate beyond the suggested response length.

3) Scoring system and evaluation criteria

 The 5-Point scoring grid was created to ensure consistency during the evaluation. This grid was
developed prior to the close of the RFSQ by a committee of subject matter experts, alongside
PriceWaterhouseCoopers LLP and Procurement. This grid was further reviewed by an
independent fairness monitor engaged to ensure the guidelines were developed in a fair and
open manner, and that they did not contain hidden evaluation criteria.
 Following a review of the evaluation notes, the PRC can confirm that the evaluation was
completed in alignment with the established criteria and guidelines.
 Prior to completion of the individual proposal evaluations, the fairness monitor also reviewed
the evaluation guidelines, including evaluator responsibilities and evaluation methodology.

4) Conflict of interest

 Throughout the process, the Province followed a rigid conflict of interest guideline process that
required participants in the RFSQ (development, review and evaluation) to declare any potential
conflicts prior to and during the procurement process.
 1660 Hollis Street,
5th Floor Centennial Bldg
Halifax, Nova Scotia B3J 1V7

 Only one (1) conflict of interest question was raised, before the deadline for questions closed,
during the RFSQ process.
 Additionally, the fairness monitor was present and monitored the various aspects of the
technical evaluation consensus process. The fairness monitor provided assurance to the PRC
that the technical evaluation consensus process was undertaken consistent with the evaluation
procedures outlined in the RFSQ and that the results reflected the consensus of all members of
each evaluation committee. The fairness monitor has confirmed that no fairness concerns were
observed.
 Lastly, the PRC did review the evaluation notes and scores for the RFSQ, and can confirm that no
bias or preference was detected.

5) Any other aspect of the complaint deemed relevant by the PRC

 Respondent’s reference evaluations were based on the information provided in the proposal.
Contacting of references was not required by the evaluation team.

 No inquiries were received during the question period directly from the Respondent, including
questions regarding the scoring system, conflict of interest or response length. Had inquiries
been received prior to the deadline for questions these could have been addressed during the
process.

 As part of the claim that the financial model identified by the Respondent in their unsolicited
proposal was divulged during a “Let’s Talk Informatics” (October 27, 2016 – AllScript), we
reviewed the presentation which was recorded and confirmed that this information was not
present.

 As noted in the complaint, the consensus scores were captured not by average but by true
agreement of the evaluation team on a score. Due to the allegations, the PRC reviewed the
scores provided for the individuals identified in the complaint as having shown bias and
recalculated the scores by average with and without the noted individuals. When averaging was
used, the PRC can confirm that the Respondent’s score would have been adversely affected in
both cases, where the scores were included as well as when they were removed.

Following this investigation, Procurement has determined that any potential conflict of interest issues
relating to the procurement process have been addressed and that all respondents were treated in a
fair, open and transparent manner, therefore the RFSQ will not be canceled.
 1660 Hollis Street,
5th Floor Centennial Bldg
Halifax, Nova Scotia B3J 1V7

Thank you again for your interest in this process.

Sincerely,

Chris Mitchell
Executive Director Procurement
Chief Procurement Officer
Chris.Mitchell@novascotia.ca