SKL2523: Information Retrieval

Lecture 11: Issues and Challenges

Issues and challenges

Multimedia IR:
- There are many fields of work that require access to
non-textual information, for example medical
professionals need access to x-rays, architects to building
plans, ornithologist to birdcalls, estate agents to property
photographs, etc.
- Multimedia information systems have been developed
to handle information contained in more than one
medium such as text, numeric data, graphics, image,
audio and video.
- Multimedia IR, especially content-based retrieval, is a
very complex area, and compared with the history of text
retrieval, multimedia IR is relatively new.

Multimedia IR
Users can now search for audio and images through
many web search engines and digital libraries.
Many multimedia retrieval systems are available but
most of them are research prototypes.
Most current research in this area is concerned with:
- images and audio retrieval: indexing using visual
and/or audio features.
- speech and music IR: research is progressing as the
current retrieval systems work well up to a certain level.
- video retrieval: a very complex area since it involves
processing a huge volume of audio, speech, natural
language and moving images.

Multimedia IR: speech

Speech retrieval:
- the conventional tools for capturing and playing speech
and audio information has a common problem: the
recorded speech or audio material has to be listened to
sequentially. Audio equipment provides mechanisms for
going forward and backward but it is difficult to retrieve
a particular segment of a speech from a long recorded
voice.
- conventional text retrieval technique may be applied to
voice retrieval easily if the spoken audio documents can
be transcribed.
- the Automatic Speech Recognition (ASR) system which
uses Hidden Markov Models (HMM) can be used for
speech retrieval.

Multimedia IR: speech

Speech retrieval:
- HMM is a statistical representation of a speech and
once a set of HMMs is generated, query speech can be
matched to find the most likely model sequence of the
recognized words.
- this approach has been found successful but one of the
major problems of ASR system is their limited accuracy,
time consuming and expensive for processing large
volumes of speech.
- the Video Mail Retrieval project at Cambridge
University with Olivetti Research Laboratory is
developing retrieval methods based on spotting
keywords in the audio soundtrack of video messages.

Multimedia IR:music
Music retrieval:
Music information consists of seven facets:
- Pitch: a quality of sound that is related to the frequency
- Tempo: information concerning the duration of a musical event
- Harmony: related to the attribute of music; a harmony occurs
when two or more pitches sound at the same time
- Timbre: an attribute related to the tone, which brings about the
aural distinction between a note played by two different instruments
- Editing: related to the performance instructions such as fingering,
ornamentation, articulation, and so on.
- Text: related to the lyrics, symphonies, and so on
- Bibliography: information about the composer, performer, title of
the piece, publisher, and so on.

Multimedia IR: music

Music retrieval:
Downie identifies two major types of music IR
systems:
- analytic or production systems, which are
intended for musicologists, music theorists,
music composers and music engravers; these
systems focus on a number of facets of music.
- locating music information retrieval systems,
which are concerned with access to musical
works; in addition to the bibliographic keys,
these retrieval systems use timbre and harmonic
features of music.

Multimedia IR: music

Music retrieval:
- Most music information retrieval systems such as those
provided by the search engines, use text-based retrieval
techniques, eg Alta Vista music search allows to search
by the name of the artist, title of the song, and also by file
types, such as MP3, WAV, Windows Media, Real or other
file types.
- The New Zealand Digital Library allows music
information retrieval by particular notes and keyword
and title. Users can search for particular notes and/or
words that appear in the music document from the
search page.

Multimedia IR: image

Image retrieval:
Areas of application:
- crime application, in matching fingerprints, matching faces,
scanning images from videos, for instance from closed circuit TV
- medicine and health, in analyzing images of health and clinical
data, such as x-rays, scans
- fashion design and graphic arts
- publishing and advertising
- engineer and architecture
- historical studies
- trademarks
- science, for instance astronomy, geography
- defence and military studies
- education and training
- entertainment.

Multimedia IR: image

Image retrieval queries:
- Image retrieval can be based on metadata (such as the creator,
date or location), associated text including the human-assigned
descriptors, or image characteristics like color, texture, shape and so
on.
- Typical image queries may include the following:
- display illustrations that may or may not be described properly in
words, eg, show me all the images of butterflies with a particular
texture of colour on the wings or show me a picture of sunset on a
golden beach
- display all the radiology images of patients with a particular
disease
- display images of the interior and exterior of a building
- display pictures where the King addresses the Parliament.
- show the images of all the free kicks taken by Beckham in the
2002 World Cup

Multimedia IR: image

Guidelines and tools for creating catalog records of images have
been developed over the years.
The main access points for images are through the name of the
creator(s), assigned theme or subject descriptors, keywords from
abstract, as well as medium, date, size, technical details, etc.
Creating catalog records for images is more complex than creating
records for text documents, as image records should contain
information about the creators, copyright, standards and tools used
for image capture, storage, etc and these information are not often
stated on the image object.
Search engines such as Google website, Yahoo Picture Gallery
(http:www.gallery.yahoo.com/), Lycos Multimedia Search
(http://multimedia.lycos.com/), and HotBot advanced
search(http://www.hotbot.com/) allow users to search for images
using keywords and other text-based image search facilities.
However there are difficulties of assigning appropriate keywords to
the images.

Multimedia IR: image

The process of describing images by human indexers is
an expensive and time-consuming process, and yet is
highly subjective.
Moreover, such processes cannot retrieve images based
on their visual characteristics.
In addition to the basic metadata attached to each object,
images may have a number of attributes that can be used
for retrieval:
- a combination of colour, texture, shape, etc
- a specific arrangement of objects in the image
- depiction of a particular event
- presence of one or more persons or objects
- presence of a specific location
- emotions attached to an event or a person, etc.

Multimedia IR: image

All these attributes can be used in what are known as
content-based image retrieval(CBIR) systems.
CBIR is a technique for retrieving images on the basis of
image characteristics such as colour, texture and shape.
However, different levels of abstraction are associated
with each type of image attribute.
Researchers have proposed different groups or types of
image retrieval based on the level of these abstractions.
Some CBIR techniques used are interfaces which can
compute similarity between a query image and image
objects; interfaces which offer options for usage of colour
histogram, selection from a palette, and storage of
features related to shape of objects.

Multimedia IR: images

CBIR systems need to focus on a number of issues such as:
- understanding the users information needs and informationseeking behavior
- finding effective ways of describing image contents
- extracting the desired features from the images
- developing efficient storage mechanisms for large image and video
databases
- matching the queries and image or video databases for similarity
- finding ways of combining images, audio and text for retrieval
from digital video
- developing efficient methods for accessing images or keyframes in
a video collection
- building suitable interfaces for user queries and display of output
from an image or a video collection.

Multimedia IR: CBIR systems

A number of software and tools have been developed of digital
images and video for CBIR. Some are available commercially, but
most CBIR software was developed as research projects. Some
examples:
- QBIC from IBM allows users to search large databases based on
visual image content (http://wwwqbic.almaden.ibm.com/)
- Virage: allows users to digitize, manage, retrieve and distribute
video and other rich media
(http://www.virage.com/about/fact_sheet/)
- Informedia: the second phase of the project seeks to improve the
dynamic extraction, summarization, visualization and presentation
of distributed video (http://www.informedia.cs.cmu.edu/)
- Alta Vista Photo Finder, a search engine which provide facilities
for CBIR (http://uk.altavista.com/image/default)
- Many digital libraries: British Library, UC Berkeley DL, DIG-LIB
at the University of Queensland, New Zealand Digital Library.

Standards for multimedia IR

Established in 1988, Moving Picture Experts Group (MPEG) is an
ISO working group in charge of the development of standards for
coded representation of digital audio and video. The group has
produced several standards, such as:
- MPEG-1 (coding of moving pictures and associated audio for
digital storage media), the standard used for VideoCD and MP3.
- MPEG-2 (generic coding of moving pictures and associated audio
information), the standard used for digital television set and DVD.
- MPEG-4, providing standardized technological elements enabling
the integration of the production, distribution and content access
paradigms of digital television, interactive graphics applications and
interactive multimedia.
- MPEG-7 (Multimedia Content Description Interface), the standard
for description and searching of audio and visual content.
- MPEG-21 (Multimedia Framework) still being developed.

Web IR: complexities

Web information retrieval is significantly different from traditional
text retrieval systems. Complexities of web IR:
1. Distributed nature of the web: web resources are distributed all
over the world, so complex measures are required to locate, index
and retrieve them; information sources are created using different
platforms, software and standards.
2. Size and growth of the web: the process of identifying, indexing
and retrieving become more complex as the volume of information
increases.
3. Deep vs the surface web: information resources on the web can be
accessed at two different levels and the deep web is several time
larger than the surface web.
4. Type and format of the documents: the web contains a variety of
type and formats of documents, from text to multimedia, thus
making the task of indexing and retrieval more complex.

Web IR: complexities

5. Quality of information: since anyone can publish
almost anything on the web, it is difficult to assess the
quality of information sources.
6. Frequency of changes: web pages change and also
move frequently thus difficult to keep track of the
changes in the millions of web pages and making the
necessary tracing to facilitate proper retrieval.
7. Ownership: some information can be accessed and
used freely but some require specific permission or
access rights. Identifying the rights to access is a major
challenge for web IR.

Web IR: complexities

8. Distributed users: users of the web resources may be
anyone and may be located anywhere in the world. This
imposes a significant challenge as the designer of a web
IR will have no idea about the target users, their nature,
characteristics, location, information search behavior,
etc.
9. Multiple languages: since the web is distributed all
over the world, the language of information sources as
well as users varies significantly. This diversity of
language poses a tremendous challenge for web IR.
10. Resource requirements: massive amount of resources
are required to build an effective and efficient web IR
systems.

Web IR: evaluation

Evaluation studies of web retrieval reveal that:
- users find it difficult to formulate their questions
- in general users spend very little time on searching a
given web search tool or database
- in most cases users formulate very short and simple
queries with one or two search terms and very few search
operators
- users spend very little time in looking at and deciding
the usefulness or relevance of retrieved items
- very few queries contain advanced search features.
Further studies are required to produce a clearer picture
of web retrieval tasks and user search behavior that will
help researchers build improved web search tools.

IR in digital libraries: problems and

prospects
IR in digital libraries involves a number of challenges. There are a
number of issues that need to be in developing efficient IR systems
for digital libraries:
-1. Problems caused by large databases: DL have to deal with a
number of databases that are huge in size. Blair argues that the
scaling problem is central to document retrieval, and performance
decreases rapidly as the size of the document database is multiplied.
-2. Problems caused by cross-database searching: designing systems
to support cross-database searching is a major challenge. Facilities
often used are the Z39.50 (a protocol designed for the exchange of
bibliographic data), metadata and a number of IR software
packages. However, major problems of interoperability occurred
from the inconsistencies of the implementation of MARC in library
OPACs, and also the use of various metadata formats such as
MARC and Dublin Core.

IR in digital libraries: problems and

prospects
3. Problems of multilingual IR: digital libraries should
support searching and display in multiple languages as a
traditional research library for example can handle
collection of materials in about 400 different languages.
There should be the need to exchange data between
digital libraries, whether in a single language or in
multiple languages. Hence multilingual search and
display in digital libraries should be a critical area of
research.
Sugimoto reports on a number of research projects that
are being carried out in Japan for multilingual
information access in digital libraries.

IR in digital libraries: problems and

prospects
4. Need for design integration:
- The IR system aims to match the user requirements
with the contents using the appropriate computer and
networking technologies.
- However, different layers of work involving the
organization and processing of information, user
interfaces, networking, standards and protocols, and so
on, are involved in the process.
- For effective IR to occur, all these different layers of a
system must be designed to work together, and the
people doing the designing must genuinely
communicate.

IR in digital libraries: problems

and prospects

5. Need for interactive question-answering

systems:
- building DL that are capable of providing
answers in an interactive question-answering
mode is a real challenge.
- it needs expertise from a number of fields
including IR, natural language processing,
human-computer interactions, expert
systems,etc.

Summary
Information retrieval for multimedia, web resources and
digital libraries is challenging as the problems are
multiplied by the volume, variety, format and language
of information resources coupled with the problems of
the widely varying nature and requirements of users, and
of information producers.
Users should be familiar with the basics of information
search techniques as well as with the information
retrieval features of those systems that are accessible
through internet and modern digital libraries.
Results of experimental studies on multimedia and
multilingual IR are promising, and one can expect to see
their applications in the future digital libraries.

References
Chowdhury, G.G. 2004. Introduction to modern
information retrieval. 2nd ed. London: Facet
Publishing.