Académique Documents
Professionnel Documents
Culture Documents
Lecture 10
Admin
Assignment 3 due
Last lecture
Move to Friday?
Projects
Come and see me
Scene motion
Objects in the scene moving
Overview
Removing Camera Shake
Non-blind
Blind
Slow-motion replay
Slow-motion replay
Motion of camera
=
Blurry image
Input to algorithm
Model is approximation
Assume static scene
Sharp image
Blur
kernel
Desired output
Convolution
operator
Blind vs Non-blind
Non-blind
Blind
Camera Shake is it a
convolution?
Person 2
Top
left
Top
right
Bot.
left
Bot.
right
Person 3
Person 4
Overview
Removing Camera Shake
Non-blind
Blind
f x y
f x y
Solution 1:
Solution 2:
Blur kernel
Frequency
spectrum
spectrum
Sharp Image
spectrum
Frequency
Frequency
1-D Example
Spatial convolution
frequency
multiplication
Unnatural
Image
gradie
nt
| f x y | i (xi )
x arg min
Convolution error
Derivatives
prior
+
Low
+
High
Input
Richardson-
(x) x
(x) x
0.8
spread gradients
localizes gradients
Gaussian prior
Sparse prior
Input
Richardson-
(x) x
(x) x
0.8
spread gradients
localizes gradients
Gaussian prior
Sparse prior
Image of star
Non-Blind Deconvolution
Matlab Demo
http://groups.csail.mit.edu/graphics/Co
dedAperture/DeconvolutionCode.html
Overview
Removing Camera Shake
Non-blind
Blind
Overview
Joint work with B. Singh, A. Hertzmann, S.T. Roweis & W.T.
Freeman
Original
Our algorithm
Close-up
Original
=
Blurry image
Input to algorithm
Model is approximation
Assume static scene
Sharp image
Blur
kernel
Desired output
Convolution
operator
Dual cameras
Coded shutter
Blur kernel
Blurry image
Log # pixels
Log # pixels
Parametric distribution
Log # pixels
Estimated
Estimated sharp image blur kernel
2. Image prior:
Distributio
n of
gradients
=
Input blurry image
3. Blur prior:
Positive
&
Sparse
b = blur kernel x =
b = blur kernel x =
b = blur kernel x =
1. Likelihood p(yjb; x)
y = observed image b = blur
image
x = sharp
Reconstruction constraint:
p(yjb; x) =
/
i - pixel
i N (yi jxi
ie
2
b; )
(xi b yi ) 2
22
x = sharp
Log # pixels
Q P C
p(x) = i c= 1 c N (f (xi )j0; s2
c)
x = sharp
Q P D
p(b) = j d= 1 d E(bj j d)
70
Mixture of Exponentials
60
constraint
p(b)
50
40
30
A few can be
large
20
10
0
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
No success!
Score
Pixel intensity
Not applicable to
natural images
Overview of algorithm
Input image
1. Pre-processing
2. Kernel estimation
-
Multi-scale approach
3. Image reconstruction
-
RAW
values
P. Debevec & J. Malik, Recovering High Dynamic Range Radiance Maps from Photographs, SIGGRAPH 97
Remapped
values
Preprocessing
Input image
Convert to
grayscale
Remove gamma
correction
User selects patch
from image
Bayesian
inference too slow
to run on whole
image
Infer kernel
from this patch
Initialization
Input image
Convert to
grayscale
Remove gamma
correction
User selects patch
from image
Initialize 3x3
blur kernel
Blurry patch
Convert to
grayscale
Remove gamma
correction
User selects patch
from image
Variation
al
Bayes
Initialize 3x3
blur kernel
Image Reconstruction
Input image
Convert to
grayscale
Remove gamma
correction
User selects patch
from image
Variation
al
Bayes
Initialize 3x3
blur kernel
Deblurred
image
Synthetic
experiments
Synthetic example
Sharp image
Artificial
blur trajectory
Image before
Kernel before
Image afte
Kernel afte
Image before
Kernel before
Inference scale 2
Image afte
Kernel afte
Image before
Kernel before
Inference scale 3
Image afte
Kernel afte
Image before
Kernel before
Inference scale 4
Image afte
Kernel afte
Image before
Kernel before
Inference scale 5
Image afte
Kernel afte
Image before
Kernel before
Inference scale 6
Image afte
Kernel afte
Image before
Kernel before
Image afte
Kernel afte
Comparison of kernels
True kernel
Estimated kernel
Blurry image
Matlabs deconvblind
Blurry image
Our output
Gamma correction
Tone response curve (if known)
DONT
Saturation
Jpeg artifacts
Scene motion
Color channel correlations
Real
experiments
Close-up
Original
Output
Original photograph
Our output
Blur kernel
Matlabs deconvblind
Close-up
Original
Our output
Matlabs
deconvblind
Original photograph
Our output
Blur kernel
Original image
Close-up
Close-up of image
Blur kernel
Original photograph
Our output
Blur kernel
Original image
Our output
Blur kernel
Close-up
Original image
Our output
Blur kerne
Blur kernel
Our output
Original photograph
Blur kernel
Our output
Close-up
Original image
Our output
Blur kernel
Original photograph
Our output
Blur kernel
Original photograph
Our output
Blur kernel
Close-up of bird
Original
Unsharp mask
Our output
Original photograph
Blur kernel
Our output
Image patterns
Note: blur kernels were inferred from large image
patches,
Summary
Method for removing camera shake
from real photographs
First method that can handle
complicated blur kernels
Uses natural image statistics
Non-blind deconvolution
currently simplistic
Things we have yet to model:
Correlations in colors, scales, kernel
continuity
Overview
Removing Camera Shake
Non-blind
Blind
Input Photo
Deblurred Result
Traditional Camera
Shutter is OPEN
Our Camera
Flutter Shutter
Implementation
Completely Portable
Lab Setup
Sync Function
Blurring
==
Convolution
Preserves High
Frequencies!!!
Comparison
Short Exposure
Long Exposure
Coded Exposure
Our result
Matlab Lucy
Ground Truth
Overview
Removing Camera Shake
Non-blind
Blind
Input image
Local Evidence
Proposed boundary
Result image
2
N (yjbx; )
Gaussian distributi
N (xj0; 2)
p(b; xjy) dx = k
p(yjb; x) p(x) dx
0.16
0.14
0.12
Bayes p(b|y)
p(bjy) =
0.1
0.08
0.06
0.04
0.02
0
0
10
MAP solution
Highest point on surface: argmaxb;x p(x; bjy)
0.16
0.14
Bayes p(b|y)
0.12
0.1
0.08
0.06
0.04
0.02
0
0
10
MAP solution
Highest point on surface: argmaxb;x p(x; bjy)
Variational Bayes
True Bayesian
approach not
tractable
Approximate
posterior
with simple
distribution
KL-Distance vs Gaussian
width
11
10
KL(q||p)
4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Variational
True
marginal
p(b|y)
1.5
MAP
0.5
0
0
10
0.9
Let true b = 2
0.8
Repeat:
Sample x ~ N(0,2)
p(b|y)
0.7
0.6
0.5
0.4
0.3
0.2
Sample n ~ N(0,2)
y = xb + n
0.1
0
0
10
b= y ! r x
Approximate posterior
with q(r x; b)
Assumeq(r
b= r y
p(r x; bjr y)
x; b) = q(r x)q(b)