Improving Image Retrieval Performance by Using Both Color and Texture Features

Dengsheng Zhang Gippsland School of Comp. and Info. Tech. Monash University, Churchill, Victoria 3842, Australia Email: dengsheng.zhang@infotech.monash.edu.au Abstract
Most content-based image retrievals (CBIR) use color as image features. However, image retrieval using color features often gives disappointing results because in many cases, images with similar colors do not have similar content. Color methods incorporating spatial information have been proposed to solve this problem, however, these methods often result in very high dimensions of features which drastically slow down the retrieval speed. In this paper, a method combining both color and texture features of image is proposed to improve the retrieval performance. Given a query, images in the database are firstly ranked using color features. Then the top ranked images are re-ranked according to their texture features. Results show the second process improves retrieval performance significantly. Keywords: CBIR, color, texture, image retrieval. of the image in some cases. Therefore, we provide two alternatives to user, one is the retrieval based on color features, the other is retrieval based on combined features. When the retrieval based on color fails, the user will use the other alternative which is the combined retrieval. By integrating these two alternatives, retrieval performance is improved significantly. The rest of the paper is organized as following. In Section 2, the color descriptor is presented. Section 3 briefly describes the rotation invariant Gabor texture features. Section 4 describes the indexing and retrieval process. Retrieval results and performance will be reported in Section 5, and the paper is concluded in Section 6.

2. Robust Color Features

To extract robust color features, we use the perceptually weighted histogram or PWH proposed by Lu et al [9]. Basically, the PWH is acquired by using CIEL*u*v* color space and assigning a perceptual weight to each histogram bin according to the distance between the color of the pixel and the color of the bin. We briefly describe the details of PWH in the following. The first step to extract color features is to select an appropriate color space. Several color spaces are available, such as RGB, CMYK, HSV and CIEL*u*v*. Most digital images are stored in RGB color space. However, RGB color space is not perceptually uniform, which implies that two colors with larger distance can be perceptually more similar than another two colors with smaller distance, or simply put, the color distance in RGB space does not represent perceptual color distance. In view of this drawback, CIEL*u*v* space is chosen because it is an uniform color space in terms of color distance. In order to use L*u*v* space, color values are first converted from RGB space into CIEXYZ space with a linear transform and then from CIEXYZ space into L*u*v* color space using the following transform:
L* = 116 3 Y / Yn 16 Y / Yn > 0.008856 L* = 903.3 (Y / Yn )
' u* = 13 L * (u 'un ) ' u* = 13 L * (v'vn )

1. Introduction
In recent years, Content Based Image Retrieval (CBIR) has been proposed in attempt to index image automatically and objectively [1, 2]. In CBIR, images in database are represented using such low level image features as color, texture and shape, which are extracted from images automatically. Among these low level features, color features are the most widely used features for image retrieval because color is the most intuitive feature and can be extracted from images conveniently [3, 4]. However, image retrieval using color features often gives disappointing results, because in many cases, images with similar colors do not have similar content. This is due to the global color features computed often fails to capture color distributions or textures within the image. Several methods have been proposed to incorporate spatial color information in attempt to avoid color confusion by machine [5, 6, 7, 8]. However, these methods often results in very high dimensions of features which drastically slow down the retrieval speed. In this paper, we propose a method combining both color and texture features to improve retrieval performance. We compute both the color and texture features from the images and images in the database are indexed using both types of features. During the retrieval process, given a query image, images in the database are firstly ranked using color features. Then in the second process, a number of top ranked images are selected and re-ranked according to their texture features. Because the texture features are extracted globally from the image, they are not an accurate description

u ' = 4 X /( X + 15Y + 3Z ) v' = 9Y /( X + 15Y + 3Z )

Y / Yn 0.008856

where

' un = 4 X n /( X n + 15Yn + 3Z n ) ' vn = 9Yn /( X n + 15Yn + 3Z n )

and (Xn, Yn, Zn) is the reference white in XYZ space. There are usually millions of colors in the color space, in order to build the color histogram from an image, the color space is normally quantized into a number of color bins with each bin represents a number of neighboring colors. However, in the L*u*v* space, representative colors are used instead of

quantizing each color channel by a constant step. The number of representative colors is given by the combinations (512) of the following three components in L*u*v* space [10]. These representative colors are uniformly distributed in L*u*v* space. L component u component v component 6.25 -110.875 -123.625 18.75 -66.625 -90.875 31.25 -22.375 -58.125 43.75 21.875 -25.375 56.25 66.125 7.375 68.75 110.375 40.125 81.25 154.625 72.875 93.75 198.875 105.625 In contrast to the conventional histogram building which assigns the color of each pixel to a single color bin, the PWH assigns the color of each pixel to 10 neighboring color bins based on the following weight: 1/ di wi = 1 / d1 + 1 / d 2 + ... + 1 / d10 where d = ( L L ) 2 + (u u ) 2 + (v v ) 2
i 0 i 0 i 0 i

generated from dilation and rotation of the following mother wavelet:

g ( x, y ) = 1 x2 y2 exp[ ( 2 + 2 )] exp( j 2Wx ) 2 x y 2 x y 1

where W is called the modulation frequency. After applying Gabor filters on the image with different orientation at different scale, an array of magnitudes is obtained:
E (m, n) = | G mn ( x, y ) |, m = 0, 1, , M - 1; n = 0, 1, , N - 1
x =0 y =0 P Q

and (L0, u0, v0) is the color of the pixel to be assigned, (Li, ui, vi) is the color of bin i. The use of PWH overcomes the drawback of conventional histogram methods which would in many situations assign a pixel color to a bin of a quite different color, e.g., assign a pixel color (6.4, 21, 40) to bin (18.75, 21.875, 40.125). The PWH also overcomes the drawback of having to assign two quite different colors to a same color bin, e.g., assigning pixel color (6.4, 21, 40) and (30, 21, 40) to bin (18.75, 21.875, 40.125) in conventional histogram building. As the result, PWH is much more accurate in representing the image than conventional histograms.

These magnitudes represent the energy content at different scale and orientation of the image. The main purpose of texture-based retrieval is to find images or regions with similar texture. It is assumed that we are interested in images or regions that have homogenous texture, therefore the mean mn and standard deviation mn of the magnitude of the transformed coefficients are used to represent the homogenous texture feature of the region. A feature vector f (texture representation) is created using mn and mn as the feature components [11]. Five scales and six orientations are used in common implementation and the Gabor texture feature vector is thus given by: f = (00 , 00 , 01 , 01 , , 45, 45). The similarity between two texture patterns is measured by the Euclidean distance between their Gabor feature vectors. To make the above extracted texture features robust to image rotation, a simple circular shift on the feature map is used [12].

4. Image Retrieval Using Both Color and Texture Features

In this section, image indexing and retrieval using color and combined features are described. Each image in the database is indexed using both color and texture features which have been described above. In other words, in the indexed database, images are represented by the indices of features. In the retrieval, images in the database, called target images, are ranked in descending order of similarity to the query image. A number of the top ranked images are presented to the user. The ranking of similarity is determined by the distance between the feature vector of the query image and the feature vector of the target image. For two images with features (f11, f12, , f1n) and (f21, f22, , f2n) respectively, the distance between the two images is given by

3. Rotation Invariant Texture Features

Texture is an important feature of objects in an image. Two images with different content can usually be distinguished by their texture features even when the images share similar colors. Gabor filters (or Gabor wavelets) are widely adopted to extract texture features from the images for image retrieval [11]. Basically, Gabor filters are a group of wavelets, with each wavelet capturing energy at a specific frequency and a specific direction. Expanding a signal using this basis provides a localized frequency description, therefore capturing local features/energy of the signal. Texture features can then be extracted from this group of energy distributions. The scale (frequency) and orientation tunable property of Gabor filter makes it especially useful for texture analysis. For a given image I(x, y) with size PQ, its discrete Gabor wavelet transform is given by a convolution:
* Gmn ( x, y ) = I ( x s, y t ) g mn ( s, t ) s =0 t =0 K K

d = ( f11 f 21 ) 2 + ( f12 f 22 ) 2 + ... + ( f1n f 2 n ) 2

Here, the fij can be either the color features or the texture features. Given a query image, the images are first retrieved by using PWH color features. Although PWH is a more accurate color histogram representation of images than conventional histogram, it is still a global representation of images. As the result, the retrieval result given by PWH usually reflects the overall color tone of the images, rather than the actual image content expected by the user. In this situation, texture features of the images can be used to help ranking images with similar content closer. However, the texture features captured are also global features, it may fail to retrieve similar images in some

* where, K is the filter mask size, and g mn is the complex conjugate of gmn which is a class of self-similar functions

situations. Solution to this problem can be segmenting image into regions and extracting texture features from the regions. However, segmentation is a complex problem which has no desirable result at this moment. Therefore, we let the user to decide if texture features is needed. If the color retrieval result is not satisfactory, the user then selects the retrieval using combined features. In the combined retrieval, the images in the database are first ranked using color features according to the distance between the query and the target images. Then a number of the top ranked images are selected and they are then re-ranked using their texture features.

provides users with only one alternative, therefore, the overall performance may not be improved in those approaches. From the precision-recall chart of Figure 2, it can be observed that the retrieval precision using color features has a sharp drop from 40% of recalls. However, retrieval precision using combined features is very robust and has a smooth drop of performance. Overall, the retrieval performance of combined features outperforms that of color features significantly.

5. Retrieval Experiments and Results

In the experiment, we use the database which is used by MPEG-7 for natural image retrieval test [13]. The database consists of 5,466 images from large varieties. The types of images in the database include animals, news photos, scenery photos, buildings, cartoons, flowers, landscapes, group of images from video sequence etc. Images with different orientations have also been included. A Java-based client-server retrieval framework has been developed to conduct the retrieval test. The server side is a Java application which handles online indexing and multi-user query. The client side is a Java applet which presents retrieval result and also provides useful user interface such as query by sketch, query by example, selecting query from list and navigation tools. To evaluate the retrieval performance, the common evaluation method, i.e., precision-recall pair, is used. Precision P is defined as the ratio of the number of retrieved relevant images r to the total number of retrieved images n, i.e. P = r/n. Precision P measures the accuracy of the retrieval. Recall R is defined as the ratio of the number of retrieved relevant images r to the total number m of relevant images in the whole database, i.e. R = r/m. Recall R measures the robustness of the retrieval. 30 images are randomly selected from the database as queries, they are shown in Figure 1. For each query, the precision of the retrieval at each level of the recall is obtained. This is done for all the 30 queries, the 30 retrieval results are averaged to give the final precision-recall chart of Figure 2. Among the 30 queries tested, 17 of the retrieval results have been improved significantly by using combined features. Some of the retrieval screen shots are shown in Figure 3 to demonstrate the comparison between retrieval using combined features and retrieval using color features. The screen shots are arranged in pairs, each pair is labeled as Ri-ct and Ri-c (i = 1, 2, , 6), where Ri-ct stands for retrieval using combined features and Ri-c for retrieval using color features. As can be observed, in all the cases demonstrated, the combined retrievals give much better results than retrievals using color features only. In some cases, retrievals using combined features give very good results while the same retrievals using color features fail almost completely, e.g., R2-ct,c, and R4ct,c. As the choice of retrieval using combined features is determine by users, it always helps to improve the retrieval results in our system. This is different from most of other approaches which integrates the two features into one and

Figure 1. The 30 random selected query images.

6. Conclusions
In this paper, an image retrieval method using both color features and combined features has been proposed. Experiment results show, our method outperforms the common color feature retrieval significantly. We have used the PWH color features to improve conventional histogram features. A robust texture feature which is suitable for image retrieval has also been presented in the paper. In contrast to most conventional combined approaches which may not give better performance than individual features, our approach provides users with two alternatives, i.e., retrieval using color features only and retrieval using combined features. Since we give the users control to select the type of retrieval, the improvement of retrieval performance is guaranteed. In the future, we plan to segment image automatically into homogenous texture regions using split and merging technique. By using regional features instead of global features, we will be able to improve retrieval performance of combined features further. We also plan to use more queries to test the retrieval performance.

References:
[1] G. Lu, Multimedia Database Management Systems, Artech House, 1999. [2] J. M. Martnez, MPEG-7 Overview (version 7), ISO/IEC JTC1/SC29/WG11 N4674, Jeju, March 2002. [3] W. Niblack et al. The QBIC Project: Querying Images By Content Using Color, Texture and Shape. SPIE Conf. On Storage and Retrieval for Image and Video Databases, Vol.1908, San Jose, CA, pp.173-187, 1993. [4] M. Swain and D. Ballard, Color Indexing, International Journal of Computer Vision, 7(1):11-32, 1991. [5] J. R. Simith and S. F. Chang, Querying by Color Regions Using the VisualSEEK Content-Based Visual Query System, In M. T. Maybury, ed., Intelligent Multimedia Information Retrieval, AAAI/MIT Press, 1996. [6] T. S. Chua, K. L. Tan and B. C. Ooi, Fast Signature-based Color-spatial Image Retrieval, In Proc. of IEEE

International Conf. on Multimedia Computing and Systems, pp362-369, Ontario, Canada, 1997. [7] W. Hsu, T. S. Chua and H. K. Pung, Integrated Color-spatial Approach to Content-based Image Retrieval, In Proc. of ACM Multimedia, pp305-313, 1995. [8] G. Pass, R. Zabih and J. Miller, Comparing Images Using Color Coherence Vectors, In Proc. of the 4th ACM Multimedia Conference, pp65-73, 1996. [9] G. Lu and J. Phillips, Using Perceptually Weighted Histograms for Color-Based Image Retrieval, In Proc. of International Conference on Signal Processing, pp11501153, Beijing, China, 1998. [10] B. Y. Chua, Color-based Image Retrieval Incorporating Relevance Feedback, Honors Thesis, Monash University, 2002. [11] B. S. Manjunath and W. Y. Ma, Texture Features for Browsing and Retrieval of Large Image Data, IEEE Trans. PAMI-18 (8):837-842, August, 1996. [12] D. S. Zhang and G. Lu, Content-based Image Retrieval Using Gabor Texture Features, In Proc. of First IEEE Pacific-Rim Conference on Multimedia (PCM'00), pp.392395, Sydney, Australia, December 13-15, 2000.

[13] B. S. Manjunath et al, Color and Texture Descriptors, IEEE Transactions on Circuits and Systems for Video Technology, 11(6):703-715, 2001.
Precision-Recall Chart 100 90 80 70
Precision Color-Texture Color

60 50 40 30 20 10 0 0 20 40
Recall

60

80

100

Figure 2. Average precision-recall of 30 queries.

R1-ct

R1-c

R2-ct

R2-c

R3-ct

R3-c

R4-ct

R4-c

R5-ct

R5-c R6-ct R6-c Figure 3. Sample screen shots of retrievals using color and combined features.