Computer Graphics II

- Image Processing and Low-Level Analysis -

May 12, 2003

Marcus Magnor

CG II: Image Processing Marcus Magnor

Organizational Things

• MPI computer account

– See Hendrik Lensch, room 228
– Then, see Sabine Budde, room 202
– You need a Linux account
• For those who also attend the seminar
– You also need a Windows account

CG II: Image Processing Marcus Magnor

 Course Structure
Need to understand
– Digital image acquisition 28.4.
– Camera geometry 5.5.
– Some basic 2D image processing algorithms 12.5.
– Photometric calibration 19.5.
To analyze
– 3D geometry from multiple images 26.5.
– Scan-based geometry acquisition 2.6.
– Material reflection properties 16.6.
Which we need for
– Fitting of theoretical models to measured data 23.6.
– Image-based rendering 30.6.
– Reflectance field rendering 7.7.
– Optical motion capture 14.7.

– Present your exercise work 21.7.

CG II: Image Processing Marcus Magnor

Overview

• Today
– Spatial filtering
– Edge detection
– Line detection
– Feature points
– Finding correspondences
– RANSAC
• Next time
– Photometric calibration

CG II: Image Processing Marcus Magnor

 Preliminaries

• Today’s lecture only about gray-scale images

– Luminance, green color channel

• Gonzalez, Woods “Digital Image Processing”

• www.dai.ed.ac.uk/CVonline/LOCAL_COPIES/
OWENS/

CG II: Image Processing Marcus Magnor

Spatial Filtering
 p1 p2 p3 
• Image pixel neighborhood (3x3 pixels) p
 4 p5 p 6 
 p7 p8 p 9 

 w1 w2 w3 
• Filter kernel (3x3 pixels) w
 4 w5 w6 
 w7 w8 w9 

• Filtering in image space: convolution

p = ∑ p iwi
– Performance depends on kernel size i

• In frequency space: multiplication

– Fourier transformation

CG II: Image Processing Marcus Magnor

 Low-Pass Filtering
• Eliminates high-frequency components
(noise)
• Smoothing, blurring
– Depends on filter kernel size
• Image restoration
• Sum of kernel = 1
• Filters
– Mean filter (averaging)
– Gaussian filter
– Median filter
– Order neighborhood pixels by value  1 2 1
1 
2 12 2 
24 
– Assign median pixel value to pixel
– Non-linear  1 2 1 
– Preserves high frequencies, eliminates
additive noise

CG II: Image Processing Marcus Magnor

Low-Pass Filtering
Different Kernel Sizes Mean vs. Median Filtering

CG II: Image Processing Marcus Magnor

 High-Pass Filtering

• Retains high-frequency components

– Image information
• Enhances noise
• Sum of kernel = 0  − 1 − 1 −1
1
− 1 8 −1
• Basic high-pass filter 9
 − 1 − 1 −1
• Derivative filters
– Direction-dependent
– Enhance oriented features

CG II: Image Processing Marcus Magnor

Gradient-based Edge Detection

Continuous function Digital samples

CG II: Image Processing Marcus Magnor

 Derivatic Filters / Gradient Detection

 ∂ I ∂I 
• Gradient of 2D image (vector !) ∇I =  , 
 ∂x ∂y 

• Direction of gradient  ∂I ∂I 
tan− 1  / 
 ∂y ∂x 

• Derivative filters  0 1
− 1 0 
1 0 
0 −1
   
– Robert’s Cross operator
− 1 − 1 − 1  − 1 0 1
∂I  ∂I 
=  0 0 0  =  − 1 0 1
∂y ∂x
– Prewitt operator  1 1 1   − 1 0 1

− 1 − 2 − 1  − 1 0 1
∂I  ∂I 
– Sobel operator =0 0 0  =  − 2 0 2 
∂y ∂x
 1 2 1   − 1 0 1 

CG II: Image Processing Marcus Magnor

Gradient Image Generation

• Sum horizontally and

vertically filtered image
– Gradient map
• Suppress all but local
maxima
– Non-maxima suppression
• Threshold local maxima
– Very sensitive to threshold
– Must be set manually
• Main problems
– Noise
– Ill-posedness:
What is an edge ?

CG II: Image Processing Marcus Magnor

 Gradient Images

CG II: Image Processing Marcus Magnor

Noise Suppression

• Low-pass filtering prior

to derivative calculation
– 1. Convolve with
Gaussian kernel
– 2. Convolve with
derivative operator
⇒ Convolve with derivative
of Gaussian

CG II: Image Processing Marcus Magnor

 2. Order Derivative Operators

• Maximum of 1. derivative ∂2 I ∂ 2 I
∇ 2I = +
= zero-crossing of 2. derivative ∂x 2 ∂ y2

• Laplacian of image
– 1. Smooth with Gaussian
– 2. Calculate 2. Derivative
⇒ Convolve with Laplacian
of Gaussian
⇒ Marr-Hildreth operator
– Mexican Hat Operator
• Find zero-crossings
– Convert to binary image <>0

CG II: Image Processing Marcus Magnor

Marr-Hildreth Operator

• Similar to low-level processing in retina

• Symmetric operator
– Edges in all directions are found simultaneously
• Produces closed contours
• Incorporates noise suppression
• Works for blurry edges
• Detects too many edges

CG II: Image Processing Marcus Magnor

 Marr-Hildreth Edge Detection

CG II: Image Processing Marcus Magnor

The Canny Edge Detector

• Idea
– Find maxima of partial derivative orthogonal to edge direction
– Smooth signal along edge direction
• Algorithm
– Smooth image (Convolve with Gaussian)
– Calculate partial derivatives
– For each pixel, remember direction of largest derivative
– Non-maximal suppression: keep only local peaks
– Store connected sets of edge points in lists
– Eliminate insignificant edges (thresholding hysteresis)
– Upper & lower threshold: keep pixels above upper threshold and
all connected pixels above lower threshold

CG II: Image Processing Marcus Magnor

 Line Detection

− 1 − 1 − 1
• Horizontal 2 2 2
 
− 1 − 1 − 1

− 1 2 − 1
• Vertical − 1 2 − 1
 
− 1 2 − 1

• 45° diagonals − 1 − 1 2   2 − 1 − 1
− 1 2 − 1 − 1 2 − 1
   
 2 − 1 − 1 − 1 − 1 2 

CG II: Image Processing Marcus Magnor

Canny Edge Detector

• Line features produce double edges

CG II: Image Processing Marcus Magnor

 Differential Edge Detection Schemes: Final Remarks

• Most commonly used in computer vision

• Problems
– Manual threshold selection
– Mask width ?
– Corners often missed (1D gradient too small)
– Find only step-like features (not thin lines)

CG II: Image Processing Marcus Magnor

The Hough Transform

• Find straight lines in image

• Line parameterization through pixel (x i,yi)
yi = a ⋅ xi + b
– Inifinitely many lines through (xi ,yi )
• Parameter space (a,b)
– Subdivide into accumulator cells
– Each cell corresponds to one line in image plane
• Algorithm
– Detect probable edge pixels (image gradient)
– For each edge pixel,
determine all possible accumulator cells
b = − xi ⋅ a + yi
– Round to next accumulator cell
– Increment accumulator cell counter

CG II: Image Processing Marcus Magnor

 Hough Transform

• Problem with slope-intercept parameterization

– Approach infinity for vertical lines
– Non-equidistant parameter space subdivision
• Normal representation
yi cos θ + xi sin θ = ρ
– Sinusoidal curves in (ρ,θ) parameter space
– Range –90 ° < θ < 90°

CG II: Image Processing Marcus Magnor

Hough Transform

CG II: Image Processing Marcus Magnor

 Hough Transform

CG II: Image Processing Marcus Magnor

Feature Point Detection

• Naïve approach
– 1. Detect isolated maxima − 1 − 1 − 1
− 1 8 − 1
– High-pass filter  
− 1 − 1 − 1
– 2. Thresholding
– Detects mainly pepper noise

• Intersection of two strong gradients

– Horizontal & vertical gradient image
– Strong signal in both gradient images
⇒ Harris corner detector

CG II: Image Processing Marcus Magnor

 Harris Corner Detector

Marc Pollefeys , KU Leuven

  ∂I 
2
 ∂I  ∂I  
 ∑  ∑     Corner response function
W  ∂x  W  ∂ x  ∂y  
M = (correlation over local neighborhood W)
  ∂I  ∂I   ∂I  
2

∑    ∑  ∂y  
 W  ∂x  ∂y  W    R = det M − k (traceM)
2
k = 0.04

Corners defined by local maxima of R

CG II: Image Processing Marcus Magnor

Point Feature Correspondence Matching

• Naïve approach
– Similarity in luminance / color of feature point pixel
– Many false matches
• Block-based correspondence matching
– Compare local neighborhood of feature points
– Take NxN-pixel area around feature points
– Compare feature point neighborhoods
N / 2 −1 N / 2 −1
– Error metrics SAD = ∑ ∑ I (x + i, y + j ) − I ' (x '+ i, y '+ j )
– Sum-of-Squared-Error i =− N / 2 j =− N / 2

– Sum-of-Absolute-Difference N / 2 −1 N / 2 −1
SSE = ∑ ∑ ( I (x + i , y + j ) − I '( x'+i , y '+ j ))
2
– Cross-correlation
i = −N / 2 j =− N / 2
between pixel blocks
l Only relative luminance,
C = ∑ ∑ (I ( x + i , y + j ) − Iˆ )⋅(I ' (x '+ i, y '+ j ) − Iˆ')
N / 2 −1 N / 2 −1

independent of absolute i =− N / 2 j =− N / 2
image intensity
l Î: mean image block luminance

CG II: Image Processing Marcus Magnor

 Cross Correlation

N / 2 −1 N / 2 −1
C= ∑ ∑ (I (x + i , y + j ) − I )⋅(I ' ( x'+i , y '+ j ) − I ')
i =− N / 2 j =− N / 2

0.9639 -0.3994 -0.1627 -0.3868 0.1914

-0.0533 0.7503 -0.4677 0.5115 0.7193
-0.1826 -0.3905 0.7730 0.1475 -0.7457
-0.2724 0.4878 0.1640 0.7862 0.2077

0.0835 0.5044 -0.4541 0.2802 0.9876

CG II: Image Processing Marcus Magnor

Correspondence Matching

Marc Pollefeys , KU Leuven

CG II: Image Processing Marcus Magnor

 Problems with Real Image Data

Radial Distortion Defocus, Blooming, Noise

CG II: Image Processing Marcus Magnor

The RANSAC Estimation Algorithm

Problem: erroneous point correspondences / mismatches /outliers
Given: many more measurements than degrees of freedom
Robust estimation: outliers do not degrade result
RANdom SAmple CONsensus: RANSAC

CG II: Image Processing Marcus Magnor

 RANSAC
Number N of trials necessary to include with probability p one sample
of size s without outliers when the outlier probability , is known:

Example for p=0.99:

If the outlier probability , is unknown:

CG II: Image Processing Marcus Magnor

Wrap-Up
• Spatial filtering in the image domain
• Image gradient calculation
• Edge pixel detection
• Gradient map
• Marr-Heredith
• Canny Edge Detector
• Straight line identification: Hough transform
• Feature points: Harris Corner Detector
• Correspondence Matching
• Block-based
• SAD, SSE, cross correlation
• RANSAC: robust estimation when outliers are present

CG II: Image Processing Marcus Magnor

 Assignment 2

• On web page
• Automatically create panorama from images
– Use Sobel gradient operators to implement Harris Corner
detector
– Find point features
– Determine correspondences
– Estimate affine transformations
– Use RANSAC to find optimal transformation matrices
– (Implement true projective transformation)
• Due May 19
– Show me your work starting at 3:30pm

CG II: Image Processing Marcus Magnor

The DLT Algorithm III

 xi   xi ' 
   
x i =  yi  ⇔ x i ' =  yi ' 
w   w '
 i  i

Only rows 1 and 2 are linearly independent ⇒ omit row 3

Inhomogeneous solution: set one matrix entry equal to 1 (e.g. h9)

Solve by Gaussian elimination or least-squares techniques

CG II: Image Processing Marcus Magnor

 Singular Value Decomposition (SVD)
Press et.al., “Numerical Recipes in C/C++”, Cambridge University Press

Problem: solve for x given A and b A⋅x = b

 w0 
 
w1
Singular value decomposition of A A = U VT
 ... 
 
 wN −1 
1 / w0 
 
1 / w
x = V  UT ⋅ b
1
Solving for x  ... 
 
 1 / wN−1 

• Can deal with near-singular sets of equations

• Least-squares solution to overdetermined set of linear equations

CG II: Image Processing Marcus Magnor

Automatic Homography Estimation Algorithm

CG II: Image Processing Marcus Magnor

 RANSAC and Guided Matching
268 putative matches 117 outliers

151 inliers 262 final correspondences

CG II: Image Processing Marcus Magnor

Newton Minimization

Given a (non-linear) vector relation y = f (x ) and an observation ŷ

⇒ Find vector x̂ satisfying yˆ = f (xˆ ) + eˆ for which ê is minimal

• Newton iteration (f locally linear):

∂y
f ( x0 + ∆) = f ( x 0 ) + J∆ J : Jacobian
∂x
Minimize eˆ = eˆ 0 − J∆ through linear least-squares:
• Solve normal equation J T J∆i = J T eˆ i for ∆ i at x i

• Update solution vector x i +1 = x i + ∆ i

Might end up in local minimum, or not converge at all

CG II: Image Processing Marcus Magnor

 Levenberg-Marquardt Iteration

f ( x0 + ∆) = f ( x 0 ) + J∆ J : Jacobian

Augment normal equation N ∆ = J T J∆ = J T eˆ with

1∀i = j
N ' ∆ = J Teˆ N ′ij = (1 + δ ijλ ) ⋅ N ij δ ij = 
0∀i ≠ j
Initialize λ to a small value (e.g., 0.001)
• Evaluate Newton scheme once
• If the value obtained for ∆ reduces the error, accept the increment
and divide λ by 10 before the next iteration
• If the error increases, multiply λ by 10 and solve augmented equation
again, until an increment is obtained that reduces the error
• For large λ, the method approaches a steepest descent

CG II: Image Processing Marcus Magnor