PANORAMIC IMAGING PART 1

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 1

Take Home Message

Q: What is a Panorama?

A: Wider-angle image than a normal camera can capture

Compensate for shortcomings of traditional cameras by capturing and
fusing many images

Technique:

Take N images at different directions

Deduce the image that would have been taken by a wide angle lens

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 2

Application: Virtual Reality

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 3

Real World Capture Setups

Jaunt NEO

Google Jump

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 4

Samsung
Beyond

Nokia OZO

Panorama:
virtual wide
Panorama:
Virtualangle
Wide Angle

Two lectures:

Part 1: Monoscopic Panoramas
(todays lecture)

Part 2: Stereoscopic Panoramas
(next lecture)

http://people.csail.mit.edu/fredo/Panos/

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 5

Panoramic Imaging Pipeline

Estimating
Correspondences

Warping

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 6

Blending

Panoramic Imaging Pipeline

Estimating
Correspondences

1. Detect keypoints
2. Find out which keypoints in im1
match which keypoints in im2

Warping

Blending

3. Compute transformation
between im2 and im1

5. Blend the images to create

a smooth transition

4. Transform im2 to overlap

with im1

6. Repeat for other images

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 7

Panoramic Imaging Pipeline

Estimating
Correspondences

Covered in upcoming lectures

For now: assume correspondences are known

Warping

Blending

3. Compute transformation
between im2 and im1

5. Blend the images to create

a smooth transition

4. Transform im2 to overlap

with im1

6. Repeat for other images

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 8

Warp to Align Images

Images: Levoy
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 9

How to Warp?

Slide: Levoy
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 10

Image Warping: Mathematical Representation

image filtering: change range of image
g(x) = T(f(x))
f

T
x

image warping: change domain of image

g(x) = f(T(x))
f

T
x

x
Slide: Efros

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 11

Image Warping: Mathematical Representation

image filtering: change range of image
g(x) = T(f(x))
g

image warping: change domain of image

g(x) = f(T(x))
f

T
Slide: Efros

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 12

Image Warping: Mathematical Representation

T

p' = (u', v')

p = (u, v)
Generic model

p' = T ( p)
Can we model the transformation as a matrix?

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 13

2D Transformations
Rotation

Scaling

! u' $ ! a 0 $! u $
#
&=#
&#
&
" v' % " 0 b %" v %

! u' $ ! cos
#
&=#
" v' % " sin

What about translation?

u' = u + tu
v' = v + tv

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 14

sin $! u $
&#
&
cos %" v %

Homogeneous Coordinates

Notation trick

Homogeneous coordinates (u, v, w) image coordinates (u/w, v/w)

Represent 2D points with 3 numbers

u' = u + tu
v' = v + tv

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 15

! u' $ ! 1 0 tu $! u $
&#
#
& #
&
# v' & = # 0 1 tv &# v &
# 1 & # 0 0 1 &# 1 &
"
% #"
%
&%"

Simple Perspective Projection

Project all points to z = f plane
Y

P(x,y,z)

p(u,v)

y
v

u = f .x / z
v = f .y / z

Viewpoint
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 16

! u $ ! U $ ! f
& #
#
& #
# v &~# V &=# 0
# 1 & # W & # 0
"
% "
% "
Image
coordinates

0 $! x
&#
f 0 &# y
0 1 &%#" z
0

Homogeneous
coordinates

$
&
&
&
%

Central Projection with Rotating Camera

(u, v)
(u', v')

O
Viewpoint

im1 im2
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 17

Transform between two images

with the same camera center

Which transform links (u, v) to (u, v) ?

Central Projection and Homography

Observation:
Lines in im1 map onto lines in im2
(u, v)
(u', v')

O
Viewpoint

im1 im2
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 18

Any transformation of a plane onto a

plane that preserves straight lines
must be a homography

Hartley & Zisserman, Theorem 2.10

Central Projection and Homography

H : non-singular 3x3 matrix
p' = (u', v',1)

p = (u, v,1)

p' = H. p
! wu' $ ! a b
#
& #
# wv' & = # d e
# w & # g h
"
% "

O
Viewpoint

im1 im2
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 19

$!
$
u
&#
&
&# v &
i &%#" 1 &%

c
f

Valid up to scale
So can set i = 1

Estimating Homography
a.uk + b.vk + c
u'k =
uk a + vk b + c uk u'k g vk v'k h = u'k
g.uk + h.vk +1
d.uk + e.vk + f
v'k =
uk d + vk e + f uk u'k g vk v'k h = v'k
g.uk + h.vk +1

Set up a system of linear equations

Ax = b
x = [a, b, c, d, e, f , g, h]T
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 20

Solve for Homography

Each correspondence gives 2 equations

Need at least 8 equations, but the more the better(why?)

If overdetermined, solve using least squares

2
Minimize || Ax b ||


Can be done using \ in Matlab

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 21

2D Image Transformations

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 22

Panoramic Imaging Pipeline

Estimating
Correspondences

Warping

Blending

3. Compute transformation
between im2 and im1

5. Blend the images to create

a smooth transition

4. Transform im2 to overlap

with im1

6. Repeat for other images

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 23

Transform between images

mosaic

p'

im2
im1

p' = H. p p = H 1 p'
im1 im2
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 24

Building a Mosaic

im1

im1
Bounding Box

mosaic

mosaic
im2
im1

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 25

Filling the Bounding Box

Determine coordinates of bounding box in im1 frame

Bounding Box

im2
im1
im1

Get samples from im1, possibly with interpolation

Apply homography to mosaic coordinates to go to im2 frame

Get samples from im2, possibly with interpolation

Overlay samples from im1 and im2


Can use meshgrid Matlab function to create mosaic coordinates
Can use interp2 Matlab function to get sub-pixel values
Watch out for NaN

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 26

Homography Example: 3D Pavement Drawings

http://www.julianbeever.net/

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 27

Panoramic Imaging Pipeline

Estimating
Correspondences

Warping

Blending

3. Compute transformation
between im2 and im1

5. Blend the images to create

a smooth transition

4. Transform im2 to overlap

with im1

6. Repeat for other images

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 28

Seams during Stitching

The problem

Image: Durand
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 29

Simple Linear Blending

L2

R1
im2

im1
im1

Set weight=1 if
only one im available
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 30

Smooth blending:
problem
Potential Artifacts

Smooth blending
Ghosting
Blurring

Basic reprojection

Image: Durand
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 31

Two-scale Blending
Idea: Combine smooth and abrupt transition

Split image into low and high frequency

Smooth blending for low frequencies: corrects exposure differences

Abrupt blending for high frequencies: avoids ghosting, preserves sharpness

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 32

Two-scale
Blending Result
Smooth
blending

Other methods:
Pyramid based
Gradient domain

Image: Durand
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 33

Panoramic Imaging Pipeline

Estimating
Correspondences

Warping

Blending

3. Compute transformation
between im2 and im1

5. Blend the images to create

a smooth transition

4. Transform im2 to overlap

with im1

6. Repeat for other images

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 34

Example: Stitching 3 Images

Image: Levoy
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 35

Example: Stitching 4 Images

How to avoid
this stretching?
Dont restrict
image surface
to a plane
Image: Levoy
Digital Image Processing 2015 Stanford University -- Panoramic Imaging 36

Cylindrical Panorama
Assume cylinder of unit radius
Y

P(x, y, z)

Unwraped Cylinder
p( , h)

= tan 1 (x / z)
h = y / x2 + z2

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 37

Spherical Panorama
Assume sphere of unit radius
Y

P(x, y, z)

Unwraped Sphere
p( , )

= tan 1 (x / z)
= tan 1 (y / x 2 + z 2 )

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 38

Sphere-to-plane mapping

Popular Mappings:

Equirectangular
Equal-area projection
Mercator
Cube maps

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 39

Example of Spherical Panorama

Constructed from 54 photographs

Figure [Image
8: An example
of a and
spherical
panorama
constructed
from 54
photographs.
Alignment
Stitching:
A Tutorial,
by Szeliski,
2006]

Professional panoramic photographers sometimes also use a pan-tilt head that makes it easy

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 40

Demo
https://graphics.stanford.edu/courses/cs178/applets/projection.html
youtube.com/360

Digital Image Processing 2015 Stanford University -- Panoramic Imaging 41