Vous êtes sur la page 1sur 113

# Extração de Características

## cap 4 – Trucco e Verri

Características de uma imagem

Arestas e cantos

## • Locais de mudanças significativas na intensidade da imagem

Edgedels = edge elements
Tipos de arestas
degrau(step) rampa(ramp)

cume(roof) impulso(spike)
Gráfico sem e com ruído

## f(x) f(x)+n(x) | f'(x)+n'(x) | f"(x)+n"(x)

Série de Taylor
(x 2 ) "
f ( x  x)  f ( x)  (x) f ( x) 
'
f ( x)  O(x 3 )
2

## Com x=1, f(x)=fi e f(x+x)=fi+1

1
f i 1  f i  f '
i  f "
i (a)
2
Com x=-1, f(x)=fi e f(x+x)=fi-1

1 (b)
f i 1  f i  f '
i  f "
i
2

f(x)

fi fi+1
fi-1

i-1 i i+1 x

(a-b)  f '
i  ( f i 1  f i 1 ) / 2

(a+b)  f "
i  (  f i 1  2 f i  f i 1 )
Em 2D
f x, y  f xn 1 , ym   f xn , ym  1 1
 f  
 x  x x
f ( x, y )   f 
 1
 y 
 f x, y  f xn , ym1   f xn , ym 
  1
y x  
Convolution Kernels
Laplaciano

2 f 2 f 1 4 1
 f ( x, y ) 
2
  
x 2
y 2  4  20 4 
1 
 4 1 
Laplaciano
•Sometimes we are interested only in changing magnitude without regard to the changing orientation.
•A linear differential operator called the Laplacian may be used.
•The Laplacian has the same properties in all directions and is therefore invariant to rotation in the image.

f "
i  (  f i 1  2 f i  f i 1 )

http://www.cee.hw.ac.uk/hipr/html/log.html

Finite differences

I x  I * 1  1

I 1
Iy  I * 
Khurram Hassan-Shafique
 1
Classical Operators

Prewitt’s Operator
1 0  1
1 1 1 0  1
1 1 1 1  
  1 0  1
1 1
Smooth Differentiate
1 1 1
1 1 1 1 0 0 0
1 1 1  1  
     1  1  1
Khurram Hassan-Shafique
Classical Operators

Sobel’s Operator
1 0  1 
1 1   2 0  2
 2 2 1 1  
  1 0  1
1 1
Differentiate
Smooth
1 2 1
1 2 1 1 0 
1 2 1  1  0 0 
     1  2  1
Khurram Hassan-Shafique
Detecting Edges in Image

## • Sobel Edge Detector

1 0  1 
 2 0  2 d
I
  dx
1 0  1
2
d  d 
2
*  I    I 
 dx   dy  Edges
Threshold
Image I
1 2 1
* 0  d
0 0 I
  dy
 1  2  1
Khurram Hassan-Shafique
Sobel Edge Detector
d
I
dx

d
I
dy

Khurram Hassan-Shafique
Sobel Edge Detector
2
d  d 
2

   I    I 
 dx   dy 

  Threshold  100
Khurram Hassan-Shafique
Marr and Hildreth Edge Operator

• Smooth by Gaussian x2  y2
S  G * I 1 
G  e 2 2

2 
• Use Laplacian to find derivatives

  2 2
 S 2 S 2 S
2

x y

Khurram Hassan-Shafique
Marr and Hildreth Edge Operator

 2 S   2 G * I    2G * I 
x2  y2
1 x y 2 2
 
 G  
2
2 
3 
e 2 2

2   2 

Khurram Hassan-Shafique
Marr and Hildreth Edge Operator
x2  y2
1 x y  2 2
 G  
2
2 
3 
e 2 2

2    2

Y
0.0008 0.0066 0.0215 0.031 0.0215 0.0066 0.0008
0.0066 0.0438 0.0982 0.108 0.0982 0.0438 0.0066
0.0215 0.0982 0 -0.242 0 0.0982 0.0215 X
0.031 0.108 -0.242 -0.7979 -0.242 0.108 0.031
0.0215 0.0982 0 -0.242 0 0.0982 0.0215
0.0066 0.0438 0.0982 0.108 0.0982 0.0438 0.0066
0.0008 0.0066 0.0215 0.031 0.0215 0.0066 0.0008

Khurram Hassan-Shafique
Marr and Hildreth Edge Operator
Image I  2G * I Edge Image
2
* G Zero Crossings
Detection

##  2G * I Zero Crossings

Khurram Hassan-Shafique
 1

 3

 6

Khurram Hassan-Shafique
Quality of an Edge Detector

• Robustness to Noise
• Localization
• Too Many/Too less Responses True Edge

## Poor robustness to noise Poor localization Too many responses

Khurram Hassan-Shafique
Canny Edge Detector
• Criterion 1: Good Detection: The optimal detector must minimize the
probability of false positives as well as false negatives.

## • Criterion 2: Good Localization: The edges detected must be as close as

possible to the true edges.

• Single Response Constraint: The detector must return one point only for
each edge point.

Khurram Hassan-Shafique
Hai Tao
The result
– General form of the filter (N.B. the filter is odd so h(x) = -h(-x) the following expression
is for x < 0 only)

## h( x)  e x (a1 sin x  a2 cosx)  ex (a3 sin x  a4 cosx)   1 / 2

a1  01486768717
.
  2.05220
a2  0.2087553476
  2.91540
a3  1244653939
.
  156939
.
a4  0.7912446531
 1  2
Camillo J. Taylor
Approximation
– Canny’s filter can be approximated by the derivative of a Gaussian

x2 x2
d  2 x  2
h( x )  ( e 2
) e 2
dx  2

## Canny Derivative of Gaussian

Camillo J. Taylor
Canny Edge Detector

## • Convolution with derivative of Gaussian

• Non-maximum Suppression
• Hysteresis Thresholding

Khurram Hassan-Shafique
Algorithm Canny_Enhancer
• Smooth by Gaussian
x2  y2

S  G * I G 
1
e 2 2

2 
• Compute x and y derivatives

T
  
S   S S   Sx  Sy 
T

 x y 
• Compute gradient magnitude and orientation

S  S x2  S y2

Sy
  tan 1

Sx Khurram Hassan-Shafique
Canny Edge Operator

S  G * I   G * I
T
 G G 
G  
 x y 

T
 G G 
S   *I *I
 x y 

Khurram Hassan-Shafique
Canny Edge Detector

Sx

Sy

Khurram Hassan-Shafique
Canny Edge Detector

S  S x2  S y2

S  Threshold  25
Khurram Hassan-Shafique
Algorithm Non-Maximum Suppression

We wish to mark points along the curve where the magnitude is biggest.
We can do this by looking for a maximum along a slice normal to the curve
(non-maximum suppression). These points should form a curve. There are
then two algorithmic issues: at which point is the maximum, and where is the
next one? Khurram Hassan-Shafique
Non-Maximum Suppression
• Suppress the pixels in ‘Gradient Magnitude Image’ which
are not local maximum

##  if S x, y   S x, y

 S x, y 
M  x, y    & S x, y   S x, y
 0
 otherwise
x, y
x, y 
x,y and x,y are the neighbors of x,y in S
x, y along the direction normal to an edge

Khurram Hassan-Shafique
Non-Maximum Suppression

Sy
tan θ  2
Sx 3 1

## 0 : -0.4142  tan θ  0.4142

1 : 0.4142  tan θ  2.4142 0
2 : tan θ  2.4142
3 : - 2.4142  tan θ  0.4142

Khurram Hassan-Shafique
Non-Maximum Suppression

S  S x2  S y2 M

M  Threshold  25
Khurram Hassan-Shafique
Hysteresis Thresholding

Khurram Hassan-Shafique
Hysteresis Thresholding

## • If the gradient at a pixel is above ‘High’, declare it an ‘edge pixel’

• If the gradient at a pixel is below ‘Low’, declare it a ‘non-edge-pixel’
• If the gradient at a pixel is between ‘Low’ and ‘High’ then declare it
an ‘edge pixel’ if and only if it is connected to an ‘edge pixel’ directly
or via pixels between ‘Low’ and ‘ High’

Khurram Hassan-Shafique
Hysteresis Thresholding

M M  Threshold  25

High  35
Low  15
Khurram Hassan-Shafique

Subpixel Localization
One can try to further localize the position of the edge within a pixel by analyzing the response to the
edge enhancement filter
– One common approach is to fit a quadratic polynomial to the filter response in the region of a maxima
and compute the true maximum.

y ( x )  ax 2  bx  c
y (0)  c;
y (1)  a  b  c;
y ( 1)  a  b  c
1
b  ( y (1)  y ( 1));
2
1
a  ( y (1)  y ( 1))  y (0);
2
-1 0 1 b
xmax 
2a
   
f(x,y) f ( p) f ( p  h n )  f ( p)
  lim
n h 0 h
 
f f n f n
 
y n x x y y f 
f f f   f  n
  nx 
n      nx  n y n
x n x y
 ny 

 f 
 
f f f f f   n x 
2

     n x 
n y  f 
 x   

n n n    x 
y   n y 

 y  
 
Detecting corners

## – If Ex and Ey denote the gradients of the intensity image, E(x,y), in

the x and y directions then the behavior of the gradients in a region
around a point can be obtained by considering the following matrix

 Ex   Ex 2 Ex E y 
C    E x Ey    
E E 2 
 Ey   x y Ey 

Camillo J. Taylor
Examining the matrix
– One way to decide on the presence of a corner is to look at the eigenvalues
of the 2 by 2 matrix C.
• If the area is a region of constant intensity we would expect both
eigenvalues to be small
• If it contains a edge we expect one large eigenvalue and one small one
• If it contains edges at two or more orientations we expect 2 large
eigenvalues

Camillo J. Taylor
Finding corners

## – One approach to finding corners is to find locations where the

smaller eigenvalue is greater than some threshold
– We could also imagine considering the ratio of the two
eigenvalues
Corner Analysis
– The ellipses indicate the eignvalues and eigenvectors of the C matrices
Corner Detection
• Basic idea: Find points where two edges meet—i.e., high gradient in two
directions
• “Cornerness” is undefined at a single pixel, because there’s only one
– Look at the gradient behavior over a small window
• Categories image windows based on gradient statistics
– Constant: Little or no brightness change
– Edge: Strong brightness change in single direction
– Flow: Parallel stripes
– Corner/spot: Strong brightness changes in orthogonal directions
Corner Detection:
• Intuitively, in corner windows both Ix and Iy should be high
– Can’t just set a threshold on them directly, because we want rotational invariance

• Analyze distribution of gradient components over a window to differentiate between types from previous
slide:

• The two eigenvectors and eigenvalues ¸1, ¸2 of C (Matlab: eig(C)) encode the predominant directions
and magnitudes of the gradient, respectively, within the window
• Corners are thus where min(¸1, ¸2) is over a threshold
courtesy of Wolfram
Contents
• Harris Corner Detector
– Description
– Analysis
• Detectors
– Rotation invariant
– Scale invariant
– Affine invariant
• Descriptors
– Rotation invariant
– Scale invariant
– Affine invariant
Harris Detector: Mathematics
Change of intensity for the shift [u,v]:

E (u, v)   w( x, y )  I ( x  u, y  v)  I ( x, y ) 
2

x, y

## Window Shifted Intensity

function intensity

## 1 in window, 0 outside Gaussian

Harris Detector: Mathematics
For small shifts [u,v] we have a bilinear approximation:

u 
E (u, v)  u, v  M  
v 

## where M is a 22 matrix computed from image derivatives:

 I x2 IxI y 
M   w( x, y )  2 
x, y  I x I y I y 
Harris Detector: Geometric Interpretation

u 
E (u, v)  u, v  M  
v 

 0  T u  1 0  u'
E  u vR 1  R     u ' v '     const
 0 2  v   0 2  v'

1, 2 – eigenvalues of M

(max)-1/2
(min)-1/2
1 (u' ) 2  2 (v' ) 2  const
Harris Detector: Mathematics
Classification of 2 “Edge”
image points using 2 >> 1 “Corner”
eigenvalues of M: 1 and 2 are large,
1 ~ 2 ;
E increases in all
directions

## 1 and 2 are small;

E is almost constant “Flat” “Edge”
in all directions region 1 >> 2

1
Harris Detector: Mathematics
Measure of corner response:

R  det M  k  trace M 
2

det M  12
trace M  1  2

## (k – empirical constant, k = 0.04-0.06)

Harris Detector: Mathematics
2 “Edge” “Corner”
• R depends only on R<0
eigenvalues of M
• R is large for a corner R>0
• R is negative with large
magnitude for an edge
• |R| is small for a flat
region “Flat” “Edge”
|R| small R<0
1
Algoritmo
R  det M  k  trace M 
2

## • Comparação dos gráficos

2 “Edge” “Corner”
R<0

R>0

“Flat” “Edge”
|R| small R<0
1
Algoritmo
R  det M  k  trace M 
2

## • Comparação dos gráficos

2 “Edge” “Corner”
R<0

R>0

“Flat” “Edge”
|R| small R<0
1
Algoritmo
R  det M  k  trace M 
2

## • Comparação dos gráficos

2 “Edge” “Corner”
R<0

R>0

“Flat” “Edge”
|R| small R<0
1
Harris Detector

• The Algorithm:
– Find points with large corner response function R (R > threshold)
– Take the points of local maxima of R
Harris Detector: Workflow
Harris Detector: Workflow
Compute corner response R
Harris Detector: Workflow
Find points with large corner response: R>threshold
Harris Detector: Workflow
Take only the points of local maxima of R
Harris Detector: Workflow

## Detail of image with gradient covar-

Full image iance ellipses for 3 x 3 windows
Example: Corner Detection
(for camera calibration)

courtesy of B. Wilburn
Example: Corner Detection

courtesy of S. Smith

SUSAN corners
Harris Detector: Summary

## • Average intensity change in direction [u,v] can be expressed as a bilinear

form:
u 
E (u, v)  u, v  M  
v 
• Describe a point in terms of eigenvalues of M:
measure of corner response

• A good (corner) point should have a large intensity change in all directions,
i.e. R should be large positive

R  12  k  1  2 
2
Contents
• Harris Corner Detector
– Description
– Analysis
• Detectors
– Rotation invariant
– Scale invariant
– Affine invariant
• Descriptors
– Rotation invariant
– Scale invariant
– Affine invariant
Tracking: compression of video information

## • Harris response (uses criss-cross gradients)

• Dinosaur tracking (using features)
• Dinosaur Motion tracking (using correlation)
• Final Tracking (superimposed)
Courtesy: (http://www.toulouse.ca/index.php4?/CamTracker/index.php4?/CamTracker/FeatureTracking.html)

This figure displays results of feature detection over the dinosaur test
sequence with the algorithm set to extract the 6 most "interesting"
features at every image frame.
It is interesting to note that although no attempt to extract frame-to-frame
feature correspondences was made, the algorithm still extracts the
same set of features at every frame.
This will be useful very much in feature tracking.
One More..

• Office sequence
• Office Tracking
Harris Detector: Some Properties

• Rotation invariance

remains the same

## Corner response R is invariant to image rotation

Harris Detector: Some Properties

##  Only derivatives are used => invariance

to intensity shift I  I + b
 Intensity scale: I  a I

R R
threshold

## x (image coordinate) x (image coordinate)

Harris Detector: Some Properties
• But: non-invariant to image scale!

## All points will be Corner !

classified as edges
Harris Detector: Some Properties
• Quality of Harris detector for different scale changes
-- Correspondences calculated using distance (and threshold)
-- Improved Harris is proposed by Schmid et al
-- repeatability rate is defined as the number of points
repeated between two images w.r.t the total number of
detected points.
 Repeatability rate:
# correspondences
# possible correspondences

## Imp.Harris uses derivative of Gaussian

instead of standard template used by
Harris et al.

## C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Contents
• Harris Corner Detector
– Description
– Analysis
• Detectors
– Rotation invariant
– Scale invariant
– Affine invariant
• Descriptors
– Rotation invariant
– Scale invariant
– Affine invariant
We want to:
detect the same interest points regardless of
image changes
Models of Image Change
• Geometry
– Rotation
– Similarity (rotation + uniform scale)

## – Affine (scale dependent on direction)

valid for: orthographic camera, locally planar object
• Photometry
– Affine intensity change (I  a I + b)
Contents
• Harris Corner Detector
– Description
– Analysis
• Detectors
– Rotation invariant
– Scale invariant
– Affine invariant
• Descriptors
– Rotation invariant
– Scale invariant
– Affine invariant
Rotation Invariant Detection

## C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Contents
• Harris Corner Detector
– Description
– Analysis
• Detectors
– Rotation invariant
– Scale invariant
– Affine invariant
• Descriptors
– Rotation invariant
– Scale invariant
– Affine invariant
Scale Invariant Detection
• Consider regions (e.g. circles) of different sizes around a point
• Regions of corresponding sizes (at different scales) will look the same
in both images

Fine/Low Coarse/High
Scale Invariant Detection
• The problem: how do we choose corresponding circles independently
in each image?
Scale Invariant Detection
• Solution:
– Design a function on the region (circle), which is “scale invariant” (the
same for corresponding regions, even if they are at different scales)

## Example: average intensity. For corresponding regions

(even of different sizes) it will be the same.
– For a point in one image, we can consider it as a
function of region size (circle radius)

f Image 1 f Image 2
scale = 1/2

## region size region size

Scale Invariant Detection
• Common approach:

## Observation: region size (scale), for which the maximum

is achieved, should be invariant to image scale.

## Important: this scale invariant region size is

found in each image independently!
Max. is called characteristic scale
f Image 1 f Image 2
scale = 1/2

## s1 region size/scale s2 region size/scal

Characteristic Scale
Max. is called characteristic scale
f Image 1 f Image 2
scale = 1/2

## s1 region size/scale s2 region size/scale

• The ratio of the scales, at which the extrema were found for
corresponding points in two rescaled images, is equal to the scale
factor between the images.
• Characteristic Scale: Given a point in an image, compute the function
responses for several factors sn The characteristic scale is the local
max. of the function (can be more than one).
• Easy to look for zero-crossings of 2nd derivative than maxima.
Scale Invariant Detection
• A “good” function for scale detection:
has one stable sharp peak

f f f Good !
region size region size region size

## • For usual images: a good function would be the one

with contrast (sharp local intensity change)
Scale Invariant Detection
• Functions for determining scale f  Kernel  Image
Kernels:

L   2  Gxx ( x, y,  )  G yy ( x, y,  ) 
(Laplacian)

DoG  G ( x, y, k )  G ( x, y,  )
(Difference of Gaussians)

where Gaussian

x2  y 2
 Note: both kernels are invariant to
G( x, y,  )  1
2
e 2 2
scale and rotation
Build Scale-Space Pyramid

## • All scales must be examined to identify scale-invariant features

• An efficient function is to compute the Difference of Gaussian
(DOG) pyramid (Burt & Adelson, 1983) (or Laplacian)
Resam
ple

Blur

Subtract

R
e
sa
mple

B
lu
r

S
ubtr
act
Key point localization

## • Detect maxima and minima of

difference-of-Gaussian in scale Resam
ple

Blur

Subtract

space
Scale Invariant Detectors

 Laplacian 
• Harris-Laplacian1 scale
Find local maximum of:
– Harris corner detector in space
(image coordinates)
– Laplacian in scale y

 Harris  x

## • SIFT (Lowe)2 scale

 DoG 
Find local maximum of:
– Difference of Gaussians in
space and scale y

 DoG  x

1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001
2 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004
Normal, Gaussian..

## A normal distribution in a variate with mean  and variance 2 is a

statistic distribution with probability function
Harris-Laplacian
• Existing methods search for maxima in the 3D representation of an
image (x,y,scale). A feature point represents a local maxima in the
surrounding 3D cube and its value is higher than a threshold.
• THIS (Harris-Laplacian) method uses Harris function first, then
selects points for which Laplacian attains maximum over scales.
• First, prepare scale-space representation for the Harris function. At
each level, detect interest points as local maxima in the image plane (of
that scale) – do this by comparing 8-neighborhood. (different from
plain Harris corner detection)
• Second, use Laplacian to judge if each of the candidate points found
on different levels, if it forms a maximum in the scale direction. (check
with n-1 and n+1)
Scale Invariant Detectors
• Experimental evaluation of detectors
w.r.t. scale change

Repeatability rate:
# correspondences
# possible correspondences
(points present)

K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001
Scale Invariant Detection: Summary
• Given: two images of the same scene with a large scale difference
between them
• Goal: find the same interest points independently in each image
• Solution: search for maxima of suitable functions in scale and in space
(over the image)

Methods:
1. Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over
scale, Harris’ measure of corner response over the image
2. SIFT [Lowe]: maximize Difference of Gaussians over scale and space
Contents
• Harris Corner Detector
– Description
– Analysis
• Detectors
– Rotation invariant
– Scale invariant
– Affine invariant (maybe later)
• Descriptors
– Rotation invariant
– Scale invariant
– Affine invariant
Affine Invariant Detection

• Above we considered:
Similarity transform (rotation + uniform scale)

• Now we go on to:
Affine transform (rotation + non-uniform scale)
Affine Invariant Detection
• Take a local intensity extremum as initial point
• Go along every ray starting from this point and stop when extremum of
function f is reached

## • We will obtain approximately

corresponding regions

## Remark: we search for scale

in every direction

## T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local,

Affinely Invariant Regions”. BMVC 2000.
Affine Invariant Detection
• Algorithm summary (detection of affine invariant region):
– Start from a local intensity extremum point
– Go in every direction until the point of extremum of some function f
– Curve connecting the points is the region boundary
– Compute geometric moments of orders up to 2 for this region
– Replace the region with ellipse

## T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local,

Affinely Invariant Regions”. BMVC 2000.
Affine Invariant Detection : Summary
• Under affine transformation, we do not know in advance shapes of the
corresponding regions
• Ellipse given by geometric covariance matrix of a region robustly approximates
this region
• For corresponding regions ellipses also correspond

Methods:
1. Search for extremum along rays [Tuytelaars, Van Gool]:
2. Maximally Stable Extremal Regions [Matas et.al.]
SIFT

• Introdução
• Motivação
• Visão Geral do Algoritmo
Introdução

• O que é o SIFT?
– Scale Invariant Feature Transform

David Lowe
Motivação

## • Como ele ajuda em Markless?

– Keypoints – Detecção de Texturas

by lowe
Visão Geral do Algoritmo

## • Detecção dos máximos locais no espaço de escala

– Usando a função diferença de Gaussianos (DoG)
• Localização dos Keypoint
– Calculo do subpixel do keypoint
• Calculo da Orientação
– 1 ou mais por Keypoint
• Keypoint descriptor
– Apartir do gradiente da imagem
Detecção dos máximos

## • Construção da piramide de DoG

by Brandon Green

Detecção dos máximos
Usa o DoG para calcular os keypoints
L ( x , y ,  )  G ( x, y ,  )  I ( x , y ) (def. espaço de Escala)
Detecção dos máximos

## • Procura nos 26 vizinhos o máximo local

Filtrando os Keypoints

## • Eliminação dos máximos que tem pouco contraste com sua

vizinhança
• Eliminação dos pontos fracos mal localizados (sobre
linhas)

1. Imagem inicial
2. Maximos
3. Eliminando por
contraste
4. Eliminando por
localização ruim
Cálculo da Orientação

## • Cálculos dos Keypoints relativos a orientação.

• Permite a invariança na rotação
Keypoint descriptor

## • Calculo referente a orientação

• Histograma com 8 classes
• Melhor descritor 4x4x8 = 128 classes
Keypoint matching

## • Calculo dos pares usando distancia Euclidiana

– Brute force
– kd – tree
– Best-Bin-First
• Calculo da transformação para alinhamento das imagens
– RANSAC
– Hough Transform
Normalmente uma imagem tem ~2,000 keypoints