2008 Third International Workshop on Semantic Media Adaptation and Personalization
Object Recognition by Modified Scale Invariant Feature Transform
Gul-e-Saman,
GIKI, Topi ,NWFP, Pakistan
Email:Gulesaman@gmail.com
Abstract
This paper presents a methodology for object
recognition. It relies on the extraction of distinctive
invariant image features that can be used to find the
correspondence between different views of an object
or a scene. These features are invariant to image
rotation and scaling, they have substantial robustness
to changes in viewpoint and illumination and addition
of noise. Mikolajczyk [1] have evaluated the SIFT [2]
algorithm along with other approaches and have
identified it as the most resistant to image distortions.
This paper improves on the SIFT algorithm by
modifying its descriptor and the keypoint localization
steps. The proposed technique uses the salient aspects
of image gradient in keypoints neighbourhood.
Moreover, instead of smoothed weighted histograms of
SIFT, Kernel Principal Component Analysis (KPCA)
is applied inorder to normalize the image patch.
Comparative results show that KPCA based
descriptors are more distinctive, robust to distortions
and compact. The evaluation of the technique is
performed using Recall Precision [3].
Keywords: 2D Laplacian filter, Haar wavelets, SIFT,
KPCA, PCA and Object recognition
1. Introduction
Object recognition is a fundamental part of many
computer vision problems. Object recognition
techniques using invariant features have become very
popular and have been developed in the last few years
[2, 4, 5]. Where the feature points corresponding to the
same points are extracted from different views, these
are invariant to image rotation, scaling, change in
illumination and viewpoint. The interest points are
robust to occlusion, clutter and photometrically
distinctive. These extracted features are matched
against reference features. Some of the detectors that
have been formulated are SIFT [2] and Harris detector
[6] which is based on the Eigen values of the second
978-0-7695-3444-2/08 $25.00 © 2008 IEEE
DOI 10.1109/SMAP.2008.12
S. Asif M. Gilani,
GIKI, Topi ,NWFP, Pakistan
Email:asif@giki.edu.pk,
moment matrix. Although, Harris corner detector is
invariant to translation and rotation, but not to scale.
The best performing detectors are Harris–Laplace and
Hessian-Laplace by Mikolajczyk and Schmid which
are robust and scale-invariant feature detectors with
high repeatability [7]. They use the determinant of
Hessian matrix to select the location and use Laplacian
for scale selection.
For local descriptors the interest points have to be
localized in scale and position. Generally, the interest
points are local peaks in a scale space search and
filtered to preserve those which would be most likely
to be stable after transformations. Then a description
of the interest point has to be built, this description has
to be unique, concise, invariant after undergoing
various transformations. The localization and
description of interest point algorithms are formulated
together, but they are tackled separately [1].
Mikolajczyk and Schmid have presented a comparison
of various local descriptors [1] as well as shape
context [8], PCA-SIFT [3], steerable filters [9],
complex filters [10], moment invariants [11] and
cross-correlation of different types of interest points
[6,7]. However, after experimentation it is not clear as
to which descriptors are more appropriate and how is
their performance dependent on the interest region
detectors.
This paper is organized as follows: Section2
reviews the salient features of the SIFT algorithm,
Section 3 explains the KPCA-SIFT, Section 4
highlights the evaluation methodology used, Section 5
gives results and comparisons of techniques, which is
followed by conclusion, acknowledgement and
references.
2. Review of SIFT Algorithm
The SIFT algorithm [2] has four major steps
namely, Scale space extrema detection, Keypoint
Localisation, Orientation Assignment and Keypoint
Descriptor computation.
The first step of the algorithm attempts to identify
the scales and locations that can be identified from
different views of an object or scene. This is done by
33
using a scale space kernel Gaussian. The scale space
is defined as follows:
L ( x, y , σ ) = G ( x, y , σ ) ∗ I ( x, y )
Where ∗ is the convolution operator, G ( x, y, σ ) is
a variable-scale Gaussian and I ( x, y ) is the input
image.
Difference of Gaussian is a technique used to detect
stable Keypoint locations in the scale space. The scale
space extrema D ( x, y, σ ) is located by computing the
difference of two images, one image scale k times the
other image.
D( x, y, σ ) = L( x, y , kσ ) − L( x, y, σ )
In order, to detect the local maxima and minima of
D( x, y, σ ) each point is compared with its 26
neighbours (8 neighbours at the same scale and 9
neighbours above and 9 neighbours below that scale)
in the scale space. If the value is a
minimum/maximum amongst all the 26 points then it
is an extrema.
The second step of the algorithm eliminates more
points from the keypoints by finding those that are
poorly localized on an edge or have low contrast.
Brown has developed a method [5] which fits a 3D
quadratic function to the keypoints inorder to
determine the interpolated location of the maximum,
according to his experiments this has substantially
improved the process of matching and stability. It
uses the Taylor expansion of the scale-space
function, D ( x, y , σ ) , shifted so that the keypoint is
the origin. The location of an extremum E, is given by:
− ∂ 2 D −1 ∂D
E= * (3)
∂x 2 ∂x
The point is excluded if the value of function at E is
below a threshold. It removes extrema that have low
contrast. For the elimination of an extrema due to poor
localisation, there is a large principal curvature across
the edge but a small curvature in the perpendicular
direction. The key point is rejected if the difference is
below the ratio of the largest to smallest eigenvector,
from the Hessian matrix at that location and scale.
The third step of the algorithm assigns an
orientation to the keypoints based on the image
properties. For the orientation computation; the
keypoints scale is used to select the image L, the
gradient magnitude m, is computed as:
m(x, y) = (L(x +1, y) − L(x −1, y))2 + (L(x, y +1) − L(x, y −1))2 (4)
The Orientation θ is computed as:
(1) θ (x, y) = tan ((L(x, y +1) − L(x, y −1))/(L(x + 1, y) − L(x −1, y))) (5)
An orientation histogram is formed of all the
keypoints. The highest peak in the histogram is
located, this peak and another peak within 80% of the
height of this peak is used to create a keypoint with
that orientation. Some points can be assigned multiple
orientations. A parabola is fit to 3 histogram values
that are closest to each peak inorder to interpolate the
peaks position.
The fourth step of the algorithm is to create the
(2) keypoint descriptor. The gradient information is used
to align with the orientation of the keypoint and then
weighted by a Gaussian with variance of 1.5 x
keypoint scale. A set of histograms is created over a
window that is centered on the keypoint with this data.
A set of 16 histograms is used, aligned in a 4x4 grid,
where there are 8 orientation bins, one each for the
main compass directions and the midpoints of these.
The resulting feature vector has 128 elements, this
vector is normalised to unit length and the elements
with small values are removed by thresholding. The
SIFT descriptor is unique in the following ways: 1) it
is carefully designed to avoid problems from the
boundaries, smooth changes in location, scale and
orientation do not fundamentally change the feature
vector, 2) it is compact as it represents a patch of
pixels by a 128 element vector, 3) its not explicitly
invariant to affine transformations, but it is flexible to
distortions by perspective effects. These
characteristics can be seen while comparing it with
competing algorithms [1].
3. Proposed Methodology
This section outlines and explains the proposed
scheme. It is based on the SIFT and PCA- SIFT
algorithms discussed earlier, by modifying their steps
and introducing steps that lead to better performance
and results.
The modifications suggested in the SIFT algorithm
are as follows: The cost of feature extraction has been
minimized by using a cascaded filtering approach,
where the more expensive operations are performed
only to specific locations that have been approved by
previous steps. It computes a large number of features
over the image. The number of features that are
detected and filtered after the first detection step is
very important for recognizing objects. These features
are extracted from a set of sample images. A query
34
image is matched by the comparison of these features
by computing the Euclidean distance.
The keypoint descriptors are computed by using the
Kernel Principal Component Analysis (KPCA) [12].
They are highly distinctive where the correct match of
features can be found with great accuracy. Moreover,
if the scene is much cluttered there might be instances
where a correct match is not found, which leads to
false matches. The set of correct matches can be sorted
produces a peak at the start of the change in the
intensity value and then at the end of the change.
The points that are obtained after the filtration in
this step have a certain degree of repeatability. Hence,
the number of the points that are detected after the
filtration process are fair enough to be used in the
recognition process.
50

out from the total set of matches by using the scale, 100

location and orientation of the query image. The 150

probability of getting the correct matches is greater 200

than false matches. 250

We propose a more accurate and robust Keypoint

300

350
localization and descriptor computation technique 400
(modification of step 2 and 4 of SIFT) while extrema 450

detection and orientation assignment is carried out 500

according to the SIFT algorithm. 50 100 150 200 250 300 350 400 450 500

(a)

Image 1 50

100

150

200
Scale-Space extrema detection 250

300

350

Keypoints Localization 400

using Laplacian Filter 450

500
50 100 150 200 250 300 350 400 450 500

(b)
Orientation Assignment Figure 2: Initial Extrema detection (a), Extrema
after filtration (b)

Keypoints Descriptor 3.2. Local Image Descriptor

using KPCA
Output Image
Figure 1: Proposed Algorithm
3.1. Accurate Keypoint localization
The local extrema that have been detected in the
earlier step are good candidates to become keypoints,
but they have to be localized. Local extrema are found
by using the 2-D Laplacian filter.
Laplacian filters are used to find areas of rapid
change in images, they are basically derivative filters.
As derivative filters are sensitive to noise, it is a
practice to smooth the image (e.g., using a Gaussian
filter) before applying the Laplacian (in this case) .
The Laplacian filter is a standard Laplacian of
Gaussian convolution. It is a second derivative
function designed such that to measure changes in
intensity values without being too sensitive to noise. It
In the earlier step location, scale and orientation
have been assigned to each keypoint. These
parameters give the description of the local image
region hence, providing invariance to these
parameters.
KPCA [12] was investigated as a generalization of
PCA. As PCA finds a second order correlation, KPCA
considers higher order correlations in images which
allow it to model data that has been generated by non-
Gaussian distributions.
KPCA can be computed by using the fact that PCA
can be used in the dot product matrix instead of
M N
covariance matrix. Let {xi ∈ R }i =1 represent a data
set. KPCA will first map the data in a higher
dimensional feature space F by the
function φ : R → F , and then performs PCA on
M
the mapped data. If the data matrix X is defined
as [φ ( x1 )φ ( x 2 ) ……φ ( x N )] , the covariance matrix
C is as follows:

35
 1 N
1 (6) N
C =
N
∑
i =1
φ ( x i )φ ( x i ) T =
N
XX T
y h = v hφ ( x) = ∑ u ih k ( xi , x) (10)
i =1

The data that has been mapped is centered i.e., Hence the φ image of x can be reconstructed from
φ ( x ) = 0 . The Eigen values and Eigen
N
1
N
∑i =1
i its projections onto the first H ( ≥ P) principal
vectors of the covariance matrix C can be computed components in F by using a projection operator PH
by solving the following Eigen Value problem:
H

λ u = Ku (7)
PH φ ( x) = ∑ y h v h (11)
h =1

Where K is the NxN matrix which is the dot

The kernel used is the Gaussian Kernel:
1 T
product matrix defined as K = X X where ⎛ x− y ⎞ (12)
N k ( x , y ) = exp ⎜⎜ − ⎟
⎝ 2σ 2 ⎟⎠
1 1 (8)
K ij = φ ( x i )φ ( x j ) = k (xi , x j ) 3.3. Implementation Details
N N
In order to compute the image descriptor a set of
Consider λ ≥ …… ≥ λ p to be the non-zero images is selected and the keypoints are detected, for
Eigen values of K ( P ≤ N , P ≤ M ) and each keypoint a 41x41 pixels image patch is extracted.
1 P The horizontal and vertical gradients are calculated
u , …… , u to be the Eigen vectors. Hence C will forming a vector these vectors are then put into a
have the same Eigen values as K forming a one to one matrix. The kernel function is applied and then the
correspondence between the non-zero Eigen covariance calculated. The Eigenvectors and Eigen
h
vectors {u } of K and non-zero Eigen vectors {v } of
h values are computed. The first n Eigen vectors are
selected to form the projection matrix. The projection
C i.e. v = α h Xu
h h
where α h is a normalization matrix is computed once and recorded for use.
constant. If both of the Eigen vectors have unit While matching the keypoints, location of
keypoints, scale and orientation act as the input to this
length, α h = 1 . It is assumed u h = 1 so descriptor. For each keypoint a 41x41 pixels image
λh N λh N patch is extracted at the given scale which is rotated to
that α h = 1 . its orientation. The horizontal and vertical gradients
are calculated forming a vector. This vector is
Eigenvectors in kernel space can be represented as
multiplied using the pre-computed projection matrix,
linearly independent samples. Without the loss of
which results in a KPCA-SIFT descriptor. The
generality, assumption should be taken that a set of m
keypoints of all images in the data set are found. All
linearly independent samples
pairs of keypoint descriptors from the images are
{φ ( x1 ),……, φ ( x m )}(m ≤ N ) span the space where N examined, those having Euclidean distance less than a
training samples are distributed. Therefore the ith threshold are considered a match.
Eigen vector vi can be represented as follows:
3.4. Feature Matching
⎡ α 1i ⎤
⎢ ⎥ Feature matching is performed by computing the
v i = [φ ( x1 ), …… , φ ( x m )]⎢ ⎥=Φ α (9) Euclidean distance between two feature vectors to
⎢ ⎥ m i
determine whether the two vectors correspond to the
⎢ ⎥
⎣α mi ⎦ same keypoint.

Where, α i = [α 1 i , … … , α mi ] T (i =1,……, m) is
4. Evaluation
We evaluate the algorithm on real images with
a coefficient vector. For some test data x, its hth different transformations (scaling, rotation,
principal component yh, which can be computed by the illumination change, change in viewpoint and addition
kernel function given as follows: of noise). We use the criterion proposed in [3], which

36
is based on the number of correct and false matches
obtained from image comparison. Receiver Operating
Characteristics (ROC) and Recall-Precision are
popular metrics and may be used interchangeably at
times. Both ROC and recall precision consider the fact
that the number of correct positive has to be increased
while decreasing the number of false positives. ROC is
quiet suitable for the evaluation of classifiers as the
rate of false detection is well defined [14], while recall
precision is suitable for the evaluation of detectors, as
the number of false detections relative to the total
number of detections is
accurately given by 1-precisin although, the total
number of negatives can not be determined.
Following the approach used in [2] the performance
of SIFT on matching keypoint is done as: matches for
a point in an image are found in the whole data set,
which being a detection task as the total number of
negatives is not well defined, and hence the suitable
metric is recall precision. We use recall vs. 1-precision
graphs as in [3].
The keypoints in the images are identified using the
modified SIFT algorithm. All pairs of keypoints are
evaluated. If the Euclidean distance between feature
vectors for a set of keypoints is below a threshold,
then points are considered as a match. When the two
keypoints correspond to the same location it is termed
as correct-positive. If the two keypoints come from
different locations then it is termed as a false-positive
match. The total number of positives for a dataset is
known as priori, from these we can formulate recall
and 1-precicion:
no. of correct positives
recall = (13)
total no. of positives
and
no. of false positives
1 − precision =
total no. of matches (correct/false)
(14)
The graphs of recall vs. 1-precision are generated.
5. Discussion of Results
This section outlines the experiments that have
been conducted with the implementation that was
proposed in the previous section in order to present an
evaluation as well as comparison with already
established techniques. Furthermore, evaluation is
carried out, as presented in the previous section. The
results from different transformations will also be
presented. The transformations applied were change in
illumination, rotation, change in viewpoint, blurring,
adding noise and scaling.
37
5.1. Image DataSet
The technique is evaluated on real images from a
database provided by Ke and
(http://www.cs.cmu.edu/~yke/pcasift/). The database
contains images with different transformations,
including change in viewpoint and scaling. We have
applied rotation to the images as rotated versions were
not available in the database.
The transformations are significant enough to
evaluate the performance of the proposed technique
and provide comparison with established technique as
well as other alternatives that were proposed during
the course of this task.
5.2. Experimental Results
For rotation testing the images are recorded by
rotating them at three degrees (3°, 6°…). It has been
observed that the percentage of matching increased for
6° rotation. For change in scale the images are
recorded by scaling them by a factor of 0.5, 0.75 and
1.5. For change in illumination the images are taken
by changing the illuminating factor by 50,75 and 100
(Adobe Photoshop 7.0). For change in viewpoint the
images are taken by changing the viewing direction of
image (Ke’s database). Gaussian noise has been added
(matlab). Experimenting on a larger database would
give a better analysis of the proposed technique. The
results acquired are shown in shown in figure 3.
6. Comparison of Techniques
This section outlines and elaborates on a
comparison between three techniques. Two of them
being those that were proposed by us and one being
the SIFT algorithm which so far has been considered
as the most efficient algorithm with substantial
experimental results. One of the techniques has been
outlined in the previous section. The other technique
also follows the main steps as that of the SIFT
algorithm, but the Haar wavelet was applied for the
keypoint localization.
Recent researches have proved that Haar wavelet is
a good feature for use in object recognition [15]. The
usefulness of using Haar features for the recognition
process has been studied by Leo et al. [15].
The Haar wavelet operates on data by calculating
the sums and differences of adjacent elements. We
have also used the differencing and averaging
procedure for the filtering out of extrema that were
detected in step 1.
Wavelets are good as far as scene matching is
required but to extract features that are repetitive other
techniques have to be used. Which was subsequently
replaced by the 2-D Laplacian filter, the points that are
obtained this way are repetitive to a fair extent even
for a transformed image.
7. Conclusion
This section initially highlights the results that were
acquired by using the proposed technique and in the
later half it compares three techniques, 1) SIFT, 2)
KPCA-SIFT (Haar) and 3) KPCA-SIFT (Laplace). It
is evident from the results that in the cases of rotation
and change of scale SIFT performs better, while SIFT
(Haar) also performs better than the KPCA-SIFT
(Laplace). For a change in viewpoint the KPCA-SIFT
(Laplace) surpasses the performance of SIFT and
KPCA-SIFT (Haar). For the Change in Illumination in
some cases there is an overlap between the KPCA-
SIFT (Laplace) and the SIFT while KPCA-SIFT(Haar)
performs better than both of them. Similarly in the
case addition of Gaussian noise to the image the
KPCA-SIFT (Laplace) performs in a similar way to
SIFT varying over a larger interval than SIFT but
KPCA-SIFT (Haar) does not perform well. For blurred
images, out of the three techniques only SIFT shows
some results but, if the blurring increases it fails
altogether. While for the comparison of the three
techniques the graphs have been highlighted in this
section as shown in Figure 4.
8. Future work
Future work would focus on invariant methods for
extrema detection and an efficient method for
orientation assignment. The computational burden on
the keypoint filtration step can be reduced further by
carefully selecting the initial keypoints. Besides this
techniques like kernel discriminant analysis (KDA)
can be used for the descriptor computation which is a
non-linear discriminating technique based on the
kernel technique. It can be used for extracting the non-
linear discriminating features. Coloured images can be
used for the detection of objects which would make it
more extensive but, also easier to manage and identify
objects. It can also be extended to be used for object
recognition in videos.
9. Acknowledgement
The authors acknowledge the support and help
provided by Asad Ali, Rubina Sultan, Nagina Hassan
38
and Madiha Hussain Malik. Thanks to Scott Ettinger
for the initial code of SIFT
10. References
[1] K.Mikolajczyk and C. Schmid, “A Performance
Evaluation of Local Descriptors” IEEE transactions on
Pattern Analysis and Machine Intelligence, Vol. 27, No.
10, October 2005
[2] D. Lowe, “Distinctive Image Features from Scale-
Invariant Keypoints”, International Journal of
Computer Vision, 2004
[3] Y. Ke and R. Sukthankar, “PCA-SIFT: A more
Distinctive Representation for Local Image
Descriptors”, Proc. Conference Computer Vision and
Pattern Recognition, 511-517, 2004
[4] K.Mikolajczyk and C. Schmid, “Scale & Affine
Invariant Interest Point Detectors” International Journal
of Computer Vision 60(1), 63-86,2004
[5] M. Brown and D. Lowe, “Invariant Features From
Interest Pont Groups”, BMVC 2002
[6] C. Harris and M. Stephens, A combined corner and
edge detector, Proc. 4th Alvey Vision Conference., 147-
151, Manchester, UK, 1988
[7] K.Mikolajczyk and C. Schmid, “Indexing based on
scale invariant interest points”. In: ICCV. Volume 1,
525-531,2001
[8] S. Belongie, J. Malik and J. Puzicha, “Shape Matching
and Object Recognition Using Shape Contexts”, IEEE
Transactions. Pattern Analysis and Machine
intelligence, Volume2, no. 4, 509-522, 2002
[9] W. Freeman and E. Adelson, “The Design and Use of
Steerable Filters”, IEEE transactions Pattern Analysis
and Machine intelligence, Volume13, no. 9, 891-906,
1991
[10] F. Schaffalitzky and A. Zisserman, “Multi-View
Matching for Unordered Image Sets”, Proc. 7th
European Conference Computer Vision, 414-431, 2002
[11] L. Van Gool, T. Moons, and D. Ungureanu, “Affine/
Photometric Invariants for Planar Intensity Patterns”,
Proc. 4th European Conference Computer Vision, 642-
651, 1996
[12] V. Vapnik, Statistical Learning Theory, Wiley, New
York, 1998
[13] B. Scholkopf, A. Smola, and K. -R. Muller, Nonlinear
Component Analysis as a Kernel Eigenvalue Problem,
Neural Computation, Vol. 10, n. 5, pp. 1299-1319,
1998.
[14] S. Agarwal and D. Roth, “Learning a sparse
representation for object detection”. In Proc. European
Conference on Computer Vision, 113-130, 2002
[15] M. Leo, T. D’Orazio, P. Spagnolo, A. Distante,
“Wavelet and ICA preprocessing for ball recognition in
soccer images”
 (a) (b) (c) (d) (e)
Figure 3: Matching with 0° rotation (a) Matching with -3° and 3° rotation (b) Matching with Illumination level set at
50 (c) Matching with Change in viewpoint (d) Matching with added noise (e)

1 1
SIFT SIFT
KPCA-SIFT(Laplace) KPCA-SIFT(Laplace)
0.95 KPCA-SIFT(Haar)
KPCA-SIFT(Haar)
0.95

0.9

0.9
1-Precision
1-Precision

0.85

0.8
0.85

0.75

0.8
0.7

0.65 0.75
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0 0.05 0.1 0.15 0.2 0.25
(a) Recall
(b) Recall

1 1.005
SIFT SIFT
0.9 KPCA-SIFT(Laplace) 1 KPCA-SIFT(Laplace)
KPCA-SIFT(Haar) KPCA-SIFT(Haar)
0.8
0.995
0.7
0.99
0.6
1-Precision
1-Precision

0.985
0.5
0.98
0.4
0.975
0.3
0.97
0.2

0.1 0.965

0 0.96
0 0.1
(c)
0.2 0.3 0.4
Recall
0.5 0.6 0.7 0.8 0 0.005 0.01
(d) 0.015 0.02
Recall
0.025 0.03 0.035 0.04

1 1
SIFT SIFT
0.9 KPCA-SIFT(Laplace) 0.9 KPCA-SIFT(Laplace)
KPCA-SIFT(Haar) KPCA-SIFT(Haar)
0.8 0.8

0.7 0.7

0.6 0.6
1-Precision
1-Precision

0.5 0.5

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0 0
0 0.1 0.2 0.3 (e) 0.4 0.5
Recall
0.6 0.7 0.8 0.9 1 0 0.005 0.01 (f)0.015 0.02
Recall
0.025 0.03 0.035 0.04

Figure 4(left to right): SIFT, Proposed Technique and SIFT (Haar) on matching tasks where the images have been:
rotated (a) scaled (b) Gaussian noise has been added (c) illumination has been changed (d) viewpoint has been
changed (d) blurred (f)

39