Evaluation of an information service

1
Table of Contents

Table of Contents...............................................................................2
1.0Introduction..................................................................................3
Explanation of the information service................................................3
3.0 Evaluation....................................................................................3
3.1 User feedback..............................................................................................3
3.2 Usage data...................................................................................................4
3.1 Survey responses.........................................................................................4
Conclusion.........................................................................................5
Bibliography......................................................................................6
Appendix...........................................................................................6
Sample Knowledge Bulletin email:.....................................................................6

2
1.0 Introduction

This report will explain the evaluation process for an information service.
The Information service to be considered is the fortnightly Knowledge
Bulletin email newsletter, as sent internally to all members of the
professional staff of Davis Langdon, a construction consultancy firm. The
Knowledge Bulletin will be evaluated for its utility and take-up among its
target audience. This report will focus on the proposed methodology of
this evaluation exercise.

Explanation of the information service

The Knowledge Bulletin is an internal current awareness publication that
is sent via email to all professional members of staff within Davis
Langdon. It is sent using mailchimp.com, an email marketing service.
The Knowledge Bulletin is comprised of summaries of articles in technical
publications and new library items, and the email contains links to
content that is available online. If the document is not available online, it
can be provided from the library in either hard copy or electronically by
request.

Historically, this current awareness publication was circulated as an

HTML email, but written in a very basic style. Specifically, it was written
in Microsoft Word, where style and formatting was applied and links to
documents on the internet were included, and then copied and pasted
into a new email message. In the winter 2008/9, the method of
distribution was changed to include a web-based email marketing service
called MailChimp. This service enables usage tracking, which provides a
built-in method of feedback.

3.0 Evaluation
There are three methods through which the library will be able to receive
feedback for this service: Unsolicited user feedback; usage data from
MailChimp; and a survey sent to readers with the Knowledge Bulletin.
The evaluation methods recommended in this report are primarily
qualitativerather than quantitative measurements. As long as the
Knowledge Bulletin is used by a certain portion of the professional staff of
Davis Langdon, it will have served its purpose. It will never be read on a
weekly basis by all members of staff, although the quantitative
measurements offered by MailChimp’s online evaluation tools will help to
measure if the improvements introduced in response to these efforts
have been valuable.

3.1 User feedback

The most direct form of feedback on the effectiveness of the Knowledge

3
Bulletin as an information service is that of direct user feedback in the
form of emailed replies requesting further information about one of the
articles featured. Before the move to MailChimp, this was the only
regular method to evaluate the service and to be sure that members of
staff were reading and using the Knowledge Bulletin.

Keeping a record of the number of people who reply to the Knowledge

Bulletin and request material, as well as the number of people who reply
with other feedback, provides a very unrepresentative survey of the
readers of the Knowledge Bulletin. By definition, the only people who will
reply in this manner are those who not only read the Knowledge Bulletin,
but those who find it useful enough to spend time to contact the library.
While most of the feedback received in this manner is very basic (ie,
‘please send me a copy of …’), users do occasionally include further
feedback, such as when explaining that they did not understand some
feature of formatting or style.

Because this feedback almost exclusively arrives by email, there is no

reason to record this feedback in an ongoing fashion—as emails, they are
automatically backed up until such time as they are needed for the
purpose of an evaluation exercise.

3.2 Usage data

Part of the justification for the move to distribution of the Knowledge
Bulletin using MailChimp is the variety of usage data that is included as
part of the regular package for distributing a newsletter. Included in this
data is information on:
• the number of people who opened the email1
• the number of people who clicked on each of the links
• how many times each person opened the email
• what time of day the email was opened
This information is provided by the MailChimp website both in online
format and it is possible to export the raw data in .csv format to a
spreadsheet programme such as Microsoft Excel. It is available for all
past newsletters sent via MailChip, and so as with email replies to the
newsletter, it is not necessary to record this data except for the purposes
of an evaluation exercise.

3.1 Survey responses

Neither of the previous two methods of evaluation actively solicits
recommendations on how to make the Knowledge Bulletin more useful
for its intended audience. Despite the note at end of each email that
states :
Please contact The Library if you have any questions about

1
For a number of technical reasons, this number is not always accurate.
However, the other information provided by MailChimp is accurate.

4
 items in this Knowledge Bulletin. We welcome any feedback.
there is no active attempt to solicit feedback on a regular basis. This is
probably as it should be, because the nature of the distribution of the
Knowledge Bulletin is such that most of its intended audience feel
comfortable in giving feedback when necessary. However, occasionally
it can be useful to solicit specific feedback in order to improve a service,
and that is why it would be useful to conduct a survey in order to help
with the evaluation of this service.

Probably the best and most effective way of conducting a survey would
be to include a link at the very top of the Knowledge Bulletin to a
SurveyMonkey.com survey that asks for specific feedback on different
portions of the newsletter. As part of a library survey in early 2008,
feedback was solicited on a range of library services including the
Knowledge Bulletin, but this feedback was simply based on the questions
regarding the Knowledge Bulletin in general. Specific questions to be
included in a survey evaluating the use of this particular information
service could include:
• Which sections of the Knowledge Bulletin do you read on a
regular basis? (Costs, Technical, Legal… all, none, it depends)
• Do you pay attention to the headings / sections, or do you read
the Knowledge Bulletin from top to bottom (or as far as you go)?
• Would you say that you pay attention when the Knowledge
Bulletin arrives in your inbox (a)sometimes; (b)always; (c)never?
• What do you primarily use the Knowledge Bulletin for? (a)
maintaining professional awareness in your specialist area; (b)keeping
current in other areas; (c) Interesting reading; (d) I read it because I
have to, but I don’t derive any value from it; (e)I don’t read it because
I don’t have the time; (f)I don’t read it because I don’t want to;
(g)Other/elaborate.
• How could the Knowledge Bulletin be more useful to you?

If a survey of this sort were to be included in the Knowledge Bulletin

at the top of the email for two successive newsletters, with the
addition of a note in the subject line (currently, the convention for the
subject line is ‘Knowledge Bulletin XX’ (where XX indicates the issue
number), an ‘including survey’ could be amended to the subject) to
attract attention,

Conclusion
This report has recommended three methods by which the library will
be able to evaluate the use of its fortnightly current awareness
publications, the Library Knowledge Bulletin. By regularly evaluating
how the product is used, library staff will be able to improve the
product to make it more relevant to current and potential consumers.

5
 As well as in-depth reviews that make use of all three of these
methods, more frequent and informal evaluation exercises can take
place using the information provided by MailChimp and by reading all
responses received in direct response to the Knowledge Bulletin.

Bibliography
Kuruppu, Pali U. ‘Evaluation of reference services – a review’ The Journal
of Academic Librarianship, Vol. 33, No. 3, pp 368–381.
‘CriSTAL checklist for evaluating a user study’
shef.ac.uk/scharr/elib/use.htm. Accessed 27 September, 2009.
mailchimp.com
http://www.mailchimp.com/articles/how_to_send_surveys_via_html_email/

Word count:

Appendix
Sample Knowledge Bulletin email:
KB 87 29 July, 2009

Sections
Costs Costs
International Presents the installation and life-cycle costs for commercial biomass heating
Markets systems. Systems typically have an expected service life of 15-25 years, and
Markets - Sectors can be classified based on the type of grate that controls fuel combustion.
Regional BLP Insurance; Building, 17.07.09, p.75
Technical
Sustainability Examines the progress in retrofitting the UK's existing housing stock in order
Legal to meet sustainability targets. Presents a value carbon analysis of improvements
APC & CPD in a case study, including CO2 savings and cumulative costs and savings for
CIS improvements.
Building Sustainable Design, August 2009, pp35-37

^Back to top

6
International
Studies the costs and benefits of sustainable healthcare facilities in
Australia. Notes that on average, hospitals require twice as much energy and
six times as much water per sq/m as an office building.
Davis Langdon; Health Sector Joins Green Ratings ERA, Research Report, 2009
(PDF download)

Examines the contribution to jobs that retrogreening offices in Australia will

have, and the part that this will play in an economic recovery for the country. 9%
of the country's labour force is involved in the construction sector.
Davis Langdon; Retrogreening Offices in Australia, Research Report, 2009 (PDF
download)

Considers the use of PFI frameworks within the UK and abroad. Lists top PFI
legal firms, financial advisors, lenders and construction firms. Also lists the top
10 PFI nations by value of work (led by Brazil, with the UK in 5th place) and by
number of deals (led by Spain, with the UK in 3rd place).
Construction News, 16.07.09, pp16-17

Costs for sewer, water and drain pipes in selected cities in the US and Canada.
Also includes cost and materials indices for major North American cities.
Engineering News Record, 13.07.09, pp51-52

Costs for asphalt, cement, aggregate, concrete, brick, block and lime in selected
cities in the US and Canada. Also includes cost and materials indices for major
North American cities.
Engineering News Record, 06.07.09, pp97-98

^Back to top

Markets
Market forecast. Compares the current recession with the previous three by
presenting historical figures for tender prices and for new-build output. Average
tender prices are now 10% lower than a year ago.
Davis Langdon; Building, 24.07.09, pp51-52

7
The Tracker. Notes that the activity indicator index has risen to 43 (the highest
level since 2008) but still represents a slowdown in the rate of activity. Evaluates
materials costs, along with a regional evaluation. East Anglia is the only UK
region to see a rise in construction activity.
Experian; Building, 17.07.09, pp62-63

Building intelligence. Explains that while the infrastructure and public sector
markets look promising, all other sectors have seen declines in outputs both
year-on-year and from the previous quarter. Total construction output has
dropped 16.5% from a year ago.
Experian; Building, 17.07.09, pp64-65

Business barometer. Lists the top contractors, architects, clients and QS firms
in the year to July 2009. Kier leads the monthly work won in June, and
Partnerships for Schools is the top client. Davis Langdon is the 3rd QS firm, with
116 projects worth £1,333m.
Building, 17.07.09, pp18-19

Examines the concrete industry. While decreased demand has led to a 33%
decrease in orders for ready mixed concrete in the first 3 months of 2009
compared to 2008, the industry hopes that recovery will be aided in part due to
gains in market share as concrete is repositioned as a sustainable commodity.
Contract Journal, 15.07.09, pp14-18

The latest Federation of Master Builders State of Trade Survey (Q2 2009)
records a general slowdown in the rate of decline, but negative trading conditions
remain throughout much of the industry.
State of Trade Survey, Federation of Master Builders, Q2 2009

^Back to top

Markets - Sectors
Considers the levels of work in the social housing sector. Lists the top
contractors in the sector, as well as the 30 biggest housing schemes in the UK.

8
Notes the levels of starts and completions in England, Wales, Scotland and
Northern Ireland.
Construction News, 16.07.09, pp24-26

Examines the status of construction work by supermarkets. Clients (led by

Tesco) have cut fees (by up to 40% so far this year) and renegotiated contracts
to stretch payment periods. The sector still accounts for £2.5bn in work.
Building, 17.07.09, pp24-26

Lists the top 10 clients and contractors in the hotel and leisure sector. Also
shows the value of regional planning approvals and examines the bright spots in
the sector throughout the UK.
Construction News, 16.07.09, pp22-23

Focus on healthcare property. Explains that the property market for redundant
hospital sites is not strong, and examines the market for healthcare
development.
Estates Gazette, 11.07.09, pp63-69

^Back to top

Regional
Hampshire, Dorset and Wiltshire. Property Week, 17.07.09
Scotland. Lists the top clients and contractors, number and value of approvals.
Construction News, 16.07.09, pp20-21
Surrey, Sussex and Kent. Estates Gazette, 11.07.09, pp71-79
Central London. Property Week, 24.07.09, pp53-60
Focus on architecture in Birmingham, including the Birmingham Central Library
and the Big City Plan. BD Magazine, July 2009
Staffordshire and Shropshire. Estates Gazette, 18.07.09, pp67-75

^Back to top

Technical
Considers the use of the New Rules of Measurement and how it relates to best

9
practice and cost planning procedures. Assesses the NRM against a number of
best practice metrics.
Davis Langdon; RICS Construction Journal, July-August 2009, pp26-28
Members of the RICS can view this article online at www.rics.org. This month's
RICS Construction Journal has a focus on the New Rules of Measurement. If you
have any trouble accessing this resource, please do not hesitate to contact The
Library.

Highlights the main benefits of the RICS New Rules of Measurement (NRM)
and considers the approach to risk and budget setting (cost limits) adopted by
the rules. Details the benefits in areas such as training and education, inflation
and preliminaries.
RICS Construction Journal, July-August 2009, pp8-10
Members of the RICS can view this article online at www.rics.org. This month's
RICS Construction Journal has a focus on the New Rules of Measurement. If you
have any trouble accessing this resource, please do not hesitate to contact The
Library.

Explains the alterations that will be taking place at the King's Cross train
station as it is refurbished. The £400m renovation programme, which includes
removing the green canopy and adding a platform for the East Coast Mainline.
New Civil Engineer, 09-16.07.09, pp22-24

Article considering the environmental benefits of fan convectors and how they
can help Housing Associations meet the Decent Homes Standard.
Housing Association Building and Maintenance, July 2009, pp19-20

^Back to top

Sustainability
Examines the sustainability of precast concrete paving particularly in relation to
new legislation that demands the use of sustainable drainage techniques such
as concrete block permeable paving.
Housing Association Building and Maintenance, July 2009, pp13-14

10
Looks at the effects of the Low Carbon Buildings Programme and the role
ground and air source heat pumps will play in future legislation to encourage
the uptake of micro-renewable technology.
Housing Association Building and Maintenance, July 2009, pp31-32

^Back to top

Legal
Examines the use of e-tendering as part of the procurement process in the
UK. While there are advantages to e-tendering, and some clients such as the
Olympic Development Authority favour it, the private sector has yet to fully
embrace the method.
Builder & Engineer, July 2009, p.8

Explains the use of contracts as protection from anticipated risks. Advises

using credit checks in order to initially investigate clients, and recommends
various ways to determine contractual rights if a client fails to pay or fulfil
contractual obligations.
Contract Journal, 15.07.09, p.28

^Back to top

APC & CPD

Introduces some principles in environmental building design, including tools
for designing with daylight. Provides an overview of how to calculate daylight in
a building.
Building Sustainable Design, August 2009, pp41-42

Examines the principles of variable flow systems. Provides an overview of all

aspects, including pumps, valve controls, system design and commissioning.
Recommends further reading on the subject.
Building Sustainable Design, August 2009, pp45-48

CPD/information services from the NBS Learning Channels are accessible from
http://channels.theNBS.com/davislangdon. Sign in using your username and

11
password, or visit the site to sign up.

^Back to top

CIS
BSRIA blue book, 2009
Annual data book presenting tables of technical data, building running costs, new
and forthcoming legislation, contact details for key industry organisations, contact
details for BSRIA's staff expertise, list of BSRIA members, industry statistics, key
performance indicators (KPIs) for 2009, and publications and services.
Blue Book, BSRIA, 2009

Performance and service life in the Environmental Profiles Methodology

and Green Guide to Specification
Explains the principles behind BRE’s approach to service life determination
developed for the update of the Environmental Profiles Methodology and The
Green Guide to Specification.
Information Paper 1/09, BRE, 2009

^Back to top

12