Mean Shift Algorithm

Clustering Segmentation Parameter Selection Recent Improvements Other Applications

Clustering
f x = Density estimate given by For each point repeat Find the Mean Shift vector

1 nh d

n i =1

x xi 2

xi g
i=1 n

m h x =

Translate the window k Iterate till convergence Associate the pixel with its mode

g
i =1

x xi h

x xi h

Results

Results

Results

Results

Segmentation Algorithm Overview

Image set of points in 5-D space Segments clusters of points occurring close together in 5D space Each segment set of points around a high density region Segmentation finding modes of the density

2 STAGES Discontinuity preserving smoothing (using Mean Shift) Clustering of pixels close in 5-D space

Mean Shift
Density estimate using Parzen windows given by
f= 1 nh
d n

K
i =1

Differentiating ( f )

h
n

xxi

f x = 2c g nh d 2 i =1

Mean Shift vector

x j 1=

xxi h
2
n i =1

xi g

x xi h
2

xi g
i =1

x xi h
2

x xi h

x xi h

Choice of Kernel

Properties of Kernel function Two commonly used kernels

Non-homogeneous space
K x =
C p h2 s hr

Epanechnikov kernel Gaussian kernel

ks

k
r

xs 2 hs

xr 2 hr

Discontinuity Preserving Smoothing

Algorithm

Let xi and zi be 5D input and output feature pixels For each pixel, repeat Initialize j = 1 and yi,1 = xi Compute yi,j+1 according to equation until convergence Assign zi = (xis , yi,Cr )

(Replace each pixel with its mode)

Discontinuity Preserving Smoothing

Discontinuity Preserving Smoothing

Segmentation

Segmentation

Group adjacent modes with similar range values into one set (Single linkage clustering) Hierarchical clustering techniques The clustering delineation step can be refined according to prior information and the physics of the problem.

Eliminate tiny clusters

If a label L has < M pixels,
Scan pixels in all 4 directions till we find pixels with support > M Sort the labels according to their support Merge L with the adjacent label having largest support

Effect and Choice of parameters

We have 3 parameters to play with

hs : Spatial resolution parameter hr : Range resolution parameter M : Size of smallest segment

Affects the smoothing, connectivity of segments Chosen depending on the size of the image, objects Affects the number of segments Should be kept low if contrast is low Should be chosen based on size of noisy patches

Effect and Choice of parameters

Original Image

hs = 8 , hr = 4

hs = 8 , hr = 8

hs = 16 , hr = 8

Colour Space

L*u*v* colour space Closely approximates uniform perceptual differences as the geometric distance between two colours

Results

Results

Results

Bandwidth Selection

Compromise in bias variance tradeoff (cross validation) Stablility of decomposition Bandwidth which maximizes an objective function Use of task dependent information... top-down

Anisotropic Kernel

Video Segmentation Video Collection of points in 6D space Motivation for Anisotropic kernel Use of Mahalanobis distance instead of Euclidean

Anisotropic Kernel
C s s s s r f x = k g x , x i , H i k n 1= 1
n

h Hi
s r s s

xr x

r i

g x , x , H = x x

s i

s i

s i

s T

Hi

s 1

xi x

Segmentation done in several passes s r H and h are varied in each pass i

Hi is calculated using neighborhood points Modified for long segments or static objects

Recent Improvements

Scale selection extension of MS tracking New Similarity Measure

Tracking through Scale Space

R.T Collins, Mean Shift blob tracking through Scale Space

Introduces scale information into the model

E x , s = H s H x , s w x
S X

HS = Epanechnikov kernel HX = DoG kernel Input image is convolved with this filter bank

Tracking through Scale Space

Algorithm

Hold 0 fixed and run Mean Shift tracking in spatial domain Now hold x constant and run Mean Shift tracking 0 in the across scales Iterate till convergence

Input = a weight image w(a) and x0, 0

New Similarity Measure

Larry Davis et al, Efficient Mean Shift tracking via a new similarity measure

Traditional measure Bhattacharya distance

p x , p y = 1 p x u p y u
u =1

Incorporates spatial information into the model

J I x , I y = 1 N
2 N N

w
i =1 i =1

hs k hr

x y

s i

s 2 j

x y

r i

r 2 j

I x , I y =log J I x , I y

New Similarity Measure

A new expression for Mean Shift is derived Advantages Incorporates spatial information Better performance than Bhattacharya and Kullback Leibler distance for low number of sample points and high dimensions Disadvantages Computation of model is O(N^2) instead of O(N)

Other Applications

Content based Image Retrieval Motion Characterization Foreground / Background separation Camera motion parameters