Measuring Effectiveness and Quality in Visual Resources

Libraries

Jennifer Durran

Abstract

“Non-profit organisations that are focused on themselves rather

than their customers display certain characteristics. They see
services as inherently desirable, blame customer ignorance or
lack of motivation when their services are not used, relegate
research about customers to a minor role...and assume they
have no generic competition.”
- Diane Tobin Johnson. “Focus on the library customer: revelation,
revolution or redundancy?” Library Trends. 43, Winter 1995: 323

Libraries and information service have traditionally considered

evaluation in terms of measuring the output and input of library
operations. These measures, whilst providing valuable information
for library managers, are becoming less relevant in the corporate
culture of higher education in the 1990s. Increasingly the emphasis
in public sector as well as private enterprise is on service quality
and value for money. It could be argued that academic libraries
have always been concerned with providing quality service but the
ongoing viability of a library is coming to depend upon its ability to
adjust its products and services to correspond to customer needs.

Visual resources collections which have traditionally serviced the

visual information needs of art historians in academic institutions
are facing a number of internal and external threats to their future
existence. By addressing the issues of evaluation, effectiveness, and
service quality (which to date have been given only cursory
attention) they will be better equipped to demonstrate the value of
their efforts not just to their customers but all their stakeholders.
The literature of visual resources librarianship reveals no research or
writing specifically on these issues. Valuable lessons can be drawn
from the experience of academic libraries with evaluation of their
effectiveness and service quality. This paper does not attempt to
review the extensive literature devoted to performance measures
for libraries, criteria for effectiveness, etc but to establish a context
for its implementation within visual resources collections.

© Jennifer Durran 1
Introduction
Visual resources libraries in academic departments or larger
information services units are beginning to feel the effect of the
changes induced by economic rationalist strategies in the higher
education sector. Visual arts and art history departments have been
forcibly amalgamated with other disciplines, closed down altogether
or are under constant threat of budget and program cuts. The fate
of those art-related slide collections which have subsequently been
relocated to another campus or institution, merged with other
facilities, dispersed or relegated to some remote corner and
forgotten is the subject of anecdote and rumour. Many of those
visual resources libraries whose future seems secure in the short-
term at least, struggle with reduced levels of funding, less staff,
increased workloads and the concurrent pressure to adopt new
technology. This situation is a microcosm reflecting changes taking
place in academic libraries and departments as universities seek to
redefine themselves and their mission in the late-twentieth century.

Higher education as a business and the implications for

libraries
Economic necessity has forced universities to actively adopt the
practices of business management. Administrators are seeking ways
to ensure their institution’s ongoing success in an intensely
competitive marketplace by:

• developing a particular profile or image, and vigorously

marketing their achievements and specialties
• rationalising, broadening and/or tailoring their course offerings
specifically to attract more potential students
• increasing part-time courses and distance education courses
• rebalancing the emphasis from research towards teaching,
especially at undergraduate level
• developing new methods of delivering education - via
computer, television and video
• expanding the support structure for students
• upgrading physical facilities and access to information
technology.

In short, they are doing everything possible to court prospective

students in order to increase their market share. Students are in
many instances paying high fees and understandably want the best
value for their money. They are more discerning and seek out those
institutions that will treat them best as paying customers.

In order to achieve cost savings and service delivery improvements

as well as the flexibility to respond to market needs, universities
are gradually contracting out functions such as security, catering,

© Jennifer Durran 2
cleaning, maintenance, etc and attempting to reduce their staff
numbers.
In this environment, every unit, system or process within the
organisation, including libraries and information services,
eventually becomes subject to rigorous assessment and is
supported or downsized, re-engineered, restructured, eliminated,
etc in accordance with its perceived value and contribution
towards the achievement of the institution’s new refocused mission.

These changes in university culture, particularly the focus on

customer service and the demand for increased accountability, have
enormous implications for the future of libraries, particularly
specialist collections such as visual resources. The literature of
library evaluation suggests that library managers have not been
particularly successful in establishing and demonstrating the value
of libraries to university administrators in terms of the cost-benefit
or cost-effectiveness of services or operations. There are some
library functions which are already being outsourced to private
enterprise -- cataloguing and selection, for example -- and in local
government in Victoria [Australia], libraries are being required to
competitively tender for the right to provide information services so
presumably some costing has been done in those instances . It has
been suggested that librarians have for too long relied on the
general consensus of libraries being inherently worthwhile and
deserving of support. According to Helen Barnes, “all they
[libraries] had to do to justify their existence was to be there
providing a service, any service.” (1992: 25).

Evaluation of libraries
Evaluation of libraries and information services has been defined as
the “process of identifying and collecting data about an organisation
or its specific programs, operations and/or services.” (Hernon and
Altman,1996: 15).

It is usually taken to involve five measurement concepts:

• extensiveness (quantity)
• effectiveness (a multidimensional construct measuring the
extent to which goals are met)
• efficiency (economy and appropriateness of resource
allocation)
• costing (cost-benefit, cost-effectiveness)
• quality (usually equated with effectiveness, but increasingly
coming to signify customer-focused service quality).

Traditionally, libraries have concentrated on measuring

extensiveness and efficiency. Visual resources libraries have

© Jennifer Durran 3
followed this model but in a fairly informal and unstructured way
with the data collected being used for annual reports and budgeting
purposes. The most widely used manual of visual resources
management covers the topic of evaluation in the context of annual
reports stating “periodic reports are effective means for
communication of the importance of the visual resources collection.
Such information can be an expedient method for evaluating the
efficiency of the collection, a means of reporting to upper
administration, informing users of activities, justifying budget needs
and developing and communicating long and short-term needs”
(Schuller, 1989: 49). Schuller’s emphasis on output and input
measures such as circulation figures, collection holdings, number of
users and borrowers, annual acquisitions, physical facilities and staff
size, reflects the preoccupation with these measures in libraries
generally.

Whilst there are inherent weaknesses in this type of performance

measurement, the data obtained can provide some valuable insights
into the effectiveness and impact of a program or operation. It does
not however convincingly demonstrate to senior management the
value of a library or information service; value being defined as the
interplay between price, accessibility and quality (Rust & Oliver,
1994). Clearly the perception of value is fundamental to the
continued existence of library services in an era where long-held
assumptions about what constitutes ‘good’ in the higher education
sector are being challenged, for example, minimal fees and
charges, low staff-student ratios, subsidised services, tenured
academic staff, wide range of subjects, etc.

Quality, one of the elements determining ‘value’, is a relatively

recent area of interest in the library and information sector,
especially the more holistic view being taken of quality issues such
as service, satisfaction and preferences as judged from the
perspective of the library clients, increasingly referred to in the
literature as ‘customers’. Hernon and Altman suggest that the of
evaluation of libraries on the basis of input and output measures
only caters to the needs and interests of the library profession and
the evaluation process should be widened to include all groups
which have interest in outcomes of library processes, the
“stakeholders” which may include librarians, student and academic
staff users and non-users, funders and administrators, politicians,
suppliers and others (Cram, 1996: 232). There is potential danger in
believing that the view of libraries as valuable and essential is
necessarily shared by the larger organisation and its administrators
(Calvert, 1995: 15). Defining the library in terms of holdings and its
services in terms of the various counts of use will eventually fail as a
way convincing administrators that the library needs more staff,
more funding, more resources, that is, the implication being that
bigger is necessarily better.

© Jennifer Durran 4
Threats to visual resource collections
The viability of many library programs is compromised not only by
this increased demand for accountability within universities but also
by a range of external forces. Extending Calvert’s argument
(1995:15), we could say that visual resource collections are
creations of a convergence of events and technological
developments in the late 19th and early 20th centuries. They
happened to have suited the needs for image storage, collection,
organisation and dissemination so far. They continue to exist
because their stakeholders want them to exist and support them but
there is no guarantee that they will continue to exist in the future
when many of their functions have been superseded by technology
or rendered obsolete by commercial vendors offering superior
services.

Some of the serious problems facing visual resource collections are:

• the continuing expansion of the subject boundaries of art
history to incorporate all forms of visual culture and the
proliferation of new specialisations. The resources needed to
acquire and maintain images for the field are not increasing in
proportion. The traditional goal of collecting and making
information accessible is fast becoming unfeasible as visual
resources collections struggle to build collections large enough
to meet local demands for images to support campus teaching
and research.
• a serious lack of research into the image needs and wants of
users. Visual resources librarians/ curators simply do not know
how well the programs and services they devise and provide
meet the needs and preferences of their customers.
Complacent attitudes (“everything is fine in my library”) and
reliance on commonsense approaches (“we intuitively know
what our customers want and need”) are naive and
unrealistic. (Hernon & Altman, 1996: 16)
• the increasing access to potentially unlimited numbers of
images via the Internet as museums, libraries and archives
develop online image databases directly from their collections.
Customers of visual resource collections are still at least a
semi-captive audience, but for how much longer?
• user-friendly image management software programs and
affordable scanners allows end-users to store their own
personalised image collections on their desktop computer,
access it at any time and transfer it easily to home or across
the world.
• the success of user-pays online document delivery services
suggests that users are prepared to pay if it means they can
get what they want, when they want it. Online commercial
image vendors are beginning to expand their services. It does

© Jennifer Durran 5
 not take too much imagination to see them developing
comprehensive search facilities for end users to access their
databases, find what they want, place an order and have a
digital file emailed to them within 24 hours.
• universities are converting lecture theatres and teaching
rooms for electronic delivery of teaching materials and the
increasing emphasis on alternative modes like distance
education, part-time courses, etc

These external threats are very real and indicate that visual
resource collections need to reassess their contribution to the
achievement of the organisation’s mission. They must “exploit the
central issue of changing organisational priorities and...
demonstrate more clearly the value of [their] information services”
if they are to control their own destiny and ensure their long-term
survival. They need to ask what their new role is to be based on
what they do best, what unique angle they can offer or what
competitive edge they can develop.

Service quality
Business enterprises have for several decades recognised that
quality management is fundamental to their continued success
especially during times of uncertainty. The public sector has also
changed their focus towards the quality of their customer service.
Useful lessons on coming to terms with service quality can be
gleaned from their experience as well from the service quality
theories which inform much of the librarianship literature on the
subject, particularly the work by Zeithaml, Parasuraman and Berry
(1990). Their definition of service quality has ten dimensions --
tangibles, reliability, responsiveness, courtesy, empathy,
competence, security, access, communication and understanding
the customer -- and three measurable perspectives of customer
satisfaction -- interactions with employees of the organisation,
services or products used, and the organisation as a system. Service
quality is determined as the gap between customer expectations
and the services provided.

The principles of quality management for libraries could be said to

comprise customer focus, unity of purpose, continual improvement,
teamwork, continued education and training and a change of
emphasis from passively providing library services to proactively
developing information products and providing a quality service
(Barnes, 1992: 28). These considerations require libraries to look at
what they do from an entirely new perspective, that of the
customer. Service quality is an issue separate from internal
observations of effectiveness. It cannot be conveyed by output and
other performance measures but as yet the librarianship profession

© Jennifer Durran 6
has failed to decide what exactly constitutes service quality and
how to measure it.

Addressing the service quality ‘gap’ has profound implications for

some of the fundamental assumptions that underpin much
academic library practice:
.
• goals and objectives of the library are aligned with those of
the university
Academic libraries (and visual resource collections) would
state that they exist to help the university achieve its goals of
teaching and research, however, the ways in which
“information is selected, acquired, stored, accessed and
distributed within the institution will, in large, determine the
level and success of teaching, scholarship and research” (ULS,
1989: 680). Hernon and Altman (1996: 1) suggest that
measures of the library’s contribution in this area could be
extended thus to visual resources collections:
- the percentage of courses using the collection and the
number of students in each course
- the percentage of students using the collection for the
preparation of papers
- the percentage of students and staff who borrow from
the collection
- the number of articles published which used material
from the collection
- the number of times material from the collection is
reproduced in a publication.

However, it must be kept in mind that research has not

established a “direct causal link between use of libraries and
the achievement of academic excellence.” (Williamson &
Exon, 1996: 532).

• only librarians know what makes a good library or

information service
It is important to identify the various stakeholders, (not just
the users) and to understand how they perceive the benefits
and value they receive from the library or information service
(McIntyre, 1992:49). We know very little, for example, about
how funders or administrators or even non-users view the
library. McIntyre believes this is an area requiring substantial
market research, the results of which should be used to design
library products that satisfy stakeholders. There may be a
perception amongst librarians that attentiveness to library
customer needs results in the undervaluing of professional
skills if, for example, “high standard of cataloguing” does not
figure in the customers’ definition of quality. It does appear
that the things that frustrate users --unavailability of wanted

© Jennifer Durran 7
 materials, faulty equipment, difficulty understanding how
library materials are organised and unresponsive staff (Hernon
& Altman, 1996: 94) - are not of high priority for many
librarians when defining quality. Librarians are only one set of
stakeholders in the library and their definition of quality is not
the only valid one. Some studies have found however that
other stakeholders have similar expectations to librarians but
with different emphases (Edwards and Browne 1995, Calvert
1994).

• doing more with less is impossible

Service quality involves the strategic deployment of resources
to narrow the gap between what the organisation and the
customers expect and the service provided. (Hernon and
Altman, 1995: 46). Faced with staff cuts or budget freezes,
libraries are forced to rethink their priorities and make
tradeoffs between cost and service. Efficiency is encouraged
as resources are better matched to activities (Mithin, 1992:
92), programs are better co-ordinated and what is not 100%
necessary is considered for elimination. Realistic quantifiable
levels of service based on finite resources can be set and
monitored. McIntyre proposes that the principle of under-
promising and then over-delivering in performance can
improve the perception of value by some stakeholders
(McIntyre, 1992: 53).

• libraries have little competition

As discussed above, libraries are facing competition from
computerised, networked databases and other electronic
information services, and threats from changes in the internal
environment of higher education. By addressing Zeithaml,
Parasuraman and Berry’s ten dimensions of service quality,
libraries could develop some concept of and recognition for
their unique service offering which incorporates the library
staff, the equipment, the ambiance, etc as well as its
collection, document delivery services, Internet access, etc .
A competitive environment is accompanied by the need for
innovation, change and leadership which in turn could result in
the development of more relevant services and projects to
attract more “customers” and therefore gain more funding.

• there is no need to promote the benefits of the library

McIntyre emphasises the importance of managing the
perceptions of stakeholders in relation to the value of the
library. As mentioned above, the perception of value is vital to
the future existence of libraries yet little attention is paid to
promoting it. This may be due to the problem of determining
a commonly accepted measure of benefit that is credible to
each set of stakeholders.

© Jennifer Durran 8
© Jennifer Durran 9
Conclusion
The changing environment in which library and information services
are delivered is rendering the evaluation of library effectiveness
based solely on quantitative performance measures less useful. The
need to justify expenditure and value for money has led librarians to
turn to other ways of measuring performance such as service
quality in order to successfully indicate their contribution towards
the institution’s achievement of its customer-centred mission.
Management is increasingly seeking confirmation that library and
information services are providing a strategic contribution to the
viability of the organisation.

Visual resources librarians, as managers of specialist information

services within the higher education sector, need to consider the
issues threatening their long-term survival. Academic libraries can
provide a model for developing appropriate methods of measuring
effectiveness and service quality. The model may as yet be an
imperfect one which has us “trying to quantify the unquantifiable,
i.e., to measure the production and transmission of recorded
knowledge and its impact, using techniques that are inherently
linear and atomistic to describe a process that is neither.” (Pritchard,
1995: 494) but unless librarians try, the institution may be forced to
impose its own view of service quality on libraries in response to
their own administrative pressures and political agendas.

References

Barnes, Helen. “A public sector manager’s perspective,” in The

Value of Library and Information Services. Melbourne: CSIRO, 1992:
24-30.

Calvert, Philip J. “Library effectiveness: the search for a social

context.” Journal of Librarianship and Information Science. 26 (1),
March 1994: 15-21.

Cram, Jennifer. “Performance management, measurement and

reporting in a time of information-centred change.” Australian
Library Journal. 45 (3), August 1996: 225-238

Edwards, Susan and Mairéad Browne. “Quality in information

services: do users and librarians differ in their expectation?” Library
and Information Science Research. 17 (2), Spring 1995: 163-182.

Hernon, Peter. “Editorial: Numbers, numbers and more numbers.”

Journal of Academic Librarianship. 22 (4), June 1996: 249-250.

© Jennifer Durran 10
 Hernon, Peter and Ellen Altman. Service Quality in Academic
Libraries. Norwood, NJ: Ablex Publishing Corp, 1996.

Johnson, Diane Tobin. “Focus on the library customer: revelation,

revolution or redundancy?” Library Trends. 43, Winter 1995: 318-
325.

Kinnell, Margaret. “Quality management and library and information

services: competitive advantage for the information revolution.”
IFLA Journal. 21, 1995: 265-273.

Kovel-Jarboe, Patricia. “Quality improvement: a strategy for planned

organizational change.” Library Trends. 44(3), Winter 1996: 605-30.

McDonald, Joseph A. and Lynda Basney Micikas. Academic Libraries:

the dimensions of their effectiveness. Westport, CT: Greenwood
Press, 1994.

McIntyre, Barry. “Ensuring outcomes of value in library and

information services,” in The Value of Library and Information
Services. Melbourne: CSIRO, 1992: 42-56.

Mithin, Bernie. “A management perspective”, in The Value of Library

and Information Services. Melbourne: CSIRO, 1992: 91-95.

Nitecki, Danuta A. “Changing the concept and measure of service

quality in academic libraries.” Journal of Academic Librarianship. 22
(3), May 1996: 181-190.

Pritchard, Sarah M. “Library Benchmarking: Old Wine in New

Bottles.” Journal of Academic Librarianship. 21, 1995: 494.

Pritchard, Sarah M. “Determining quality in academic libraries.”

Library Trends. 44 (3), Winter 1996 : 572-91.

Rust, Roland T. & Richard L. Oliver. Service quality: New directions

in theory and practice. Thousand Oaks,CA: Sage, 1994.

Schuller, Nancy Shelby. Management for Visual Resources

Collections. 2nd ed. Englewood,CO : Libraries Unlimited Inc: 1989.

ULS University Library Standards Review Committee. “Standards for

university libraries: evaluation of performance.” College & Research
Libraries News. 50 (8), 1989: 680.

Van House, Nancy A., Beth T. Weil and Charles R. McClure.

Measuring Academic Library Performance: A Practical Approach.
Chicago: ALA, 1990.

© Jennifer Durran 11
Williamson, Vicki and F.C.A. Exon. “The Quality Movement in
Australian University Libraries.” Library Trends. 44 (3), Winter 1996:
526-44.

About this paper

This paper was originally published in ARLIS/ANZ News, No. 46,
March 1998, 25-31.

About the author

At the time of writing the author was Visual Arts Librarian, Monash
University, Melbourne, Australia.

© Jennifer Durran 12