C280 Computer Vision C280,ComputerVision

Prof.TrevorDarrell
trevor@eecs.berkeley.edu
Lecture10:Stereo
Roadmap
Previous: Image formation filtering local features (Texture) Previous:Imageformation,filtering,localfeatures,(Texture)
Tues:FeaturebasedAlignment
Stitchingimagestogether
Homographies,RANSAC,Warping,Blending
Globalalignmentofplanarmodels
Today: Dense Motion Models Today:DenseMotionModels
Localmotion/featuredisplacement
Parametricopticflow
Noclassesnextweek:ICCVconference
Oct6
th
:Stereo/Multiview:Estimatingdepthwithknown
i t intercamerapose
Oct8
th
:Structurefrommotion:Estimationofposeand3D
structure
Factorizationapproaches
Globalalignmentwith3Dpointmodels
Last time: Motion and Flow Lasttime:MotionandFlow
M ti ti ti Motionestimation
Patchbasedmotion
(opticflow)
Regularizationandline
processes
Parametric (global) Parametric(global)
motion
Layered motion models Layeredmotionmodels
Today: Stereo Today:Stereo
Human stereopsis & stereograms Humanstereopsis &stereograms
Epipolar geometryandtheepipolar constraint
Caseexamplewithparallelopticalaxes
Generalcasewithcalibratedcameras
Correspondencesearch
TheEssentialandtheFundamentalMatrix e sse t a a d t e u da e ta at
Multiviewstereo
Fixation, convergence Fixation,convergence
Grauman
Human stereopsis: disparity Humanstereopsis:disparity
Disparity occurswheneyes
fi t bj t th fixateononeobject;others
appearatdifferentvisual
angles g
Humanstereopsis:disparity
d=0
Disparity:d=rl=DF.
AdaptedfromM.Pollefeys
Random dot stereograms Randomdotstereograms
Julesz 1960: Do we identify local brightness Julesz1960:Doweidentifylocalbrightness
patternsbeforefusion(monocularprocess)or
after (binocular)? after(binocular)?
T i f h i i b i d b Totest:pairofsyntheticimagesobtainedby
randomlysprayingblackdotsonwhiteobjects
Random dot stereograms Randomdotstereograms
Forsyth&Ponce
Random dot stereograms Randomdotstereograms
Random dot stereograms Randomdotstereograms
FromPalmer,VisionScience,MITPress
Random dot stereograms Randomdotstereograms
When viewed monocularly, they appear random; Whenviewedmonocularly,theyappearrandom;
whenviewedstereoscopically,see3dstructure.
Conclusion:humanbinocularfusionnotdirectly
associatedwiththephysicalretinas;mustinvolvethe p y
centralnervoussystem
Imaginarycyclopeanretinathatcombinestheleft
andrightimagestimuliasasingleunit
Grauman
Autostereograms Autostereograms
Exploit disparity as depth Exploitdisparityasdepth
cueusingsingleimage
(Singleimagerandomdot ( g g
stereogram,Singleimage
stereogram)
Imagesfrommagiceye.com
Autostereograms Autostereograms
Imagesfrommagiceye.com
Stereophotographyandstereoviewers
Taketwopicturesofthesamesubjectfromtwoslightlydifferent
viewpointsanddisplaysothateacheyeseesonlyoneofthe p p y y y
images.
InventedbySirCharlesWheatstone,1838
Imagecourtesyoffisherprice.com
Grauman
http://www.johnsonshawmuseum.org
Grauman
http://www.johnsonshawmuseum.org
Grauman
PublicLibrary,StereoscopicLookingRoom,Chicago,byPhillips,1923
Grauman
http://www.well.com/~jimg/stereo/stereo_list.html
Grauman
Depth with stereo: basic idea Depthwithstereo:basicidea
ii scenepoint scenepoint
imageplane imageplane
opticalcenter opticalcenter
Source:SteveSeitz
Depth with stereo: basic idea Depthwithstereo:basicidea
BasicPrinciple:Triangulation
Gives reconstruction as intersection of two rays Givesreconstructionasintersectionoftworays
Requires
camerapose(calibration)
pointcorrespondence
Source:SteveSeitz
Cameracalibration
Extrinsic parameters:
World
frame
Intrinsic parameters:
p
CameraframeReferenceframe
Cameraframe
Imagecoordinatesrelativetocamera
Pixelcoordinates
Extrinsic params:rotationmatrixandtranslationvector
Intrinsic params:focallength,pixelsizes(mm),imagecenterpoint,
radialdistortionparameters
Wellassumefornowthattheseparametersaregiven f p g
andfixed.
Grauman
Today Today
Humanstereopsis p
Stereograms
E i l t d th i l t i t Epipolargeometryandtheepipolarconstraint
Caseexamplewithparallelopticalaxes
Generalcasewithcalibratedcameras
Stereopsis
Findingcorrespondencesalongtheepipolarline
Geometryforasimplestereosystem
First,assumingparallelopticalaxes,knowncamera
parameters (i e calibrated cameras): parameters(i.e.,calibratedcameras):
World
point point
Depthofp
imagepoint
(left)
imagepoint
(right)
Focal
length
optical
center (left)
optical
center
(right)
length
baseline
center(left)
(right)
Geometryforasimplestereosystem
Assumeparallelopticalaxes,knowncameraparameters(i.e.,
calibratedcameras).Wecantriangulatevia:
Similartriangles(p
l
,P,p
r
)and(O
l
,
P,O
r
): ,
r
)
T x x T
r l
=
+
Z f Z
l
x x
T
f Z

=
l r
x x
disparity
Grauman
Disparity example
image I(x y)
image I(x y)
Disparity map D(x y)
Disparityexample
imageI(x,y)
imageI (x ,y )
DisparitymapD(x,y)
(x y)=(x+D(x y) y) (x ,y )=(x+D(x,y),y)
Grauman
Generalcase,withcalibratedcameras
Thetwocamerasneednothaveparallelopticalaxes.
Vs.
Grauman
Stereocorrespondenceconstraints
Given p in left image, where can corresponding point Givenpinleftimage,wherecancorrespondingpoint
pbe?
Stereocorrespondenceconstraints
Given p in left image, where can corresponding point Givenpinleftimage,wherecancorrespondingpoint
pbe?
Stereocorrespondenceconstraints
Stereocorrespondenceconstraints
Geometryoftwoviewsallowsustoconstrainwherethecorresponding
pixelforsomeimagepointinthefirstviewmustoccurinthesecond
view.
epipolarplane
epipolarline epipolarline
epipolarline epipolarline
Epipolar constraint: Why is this useful? Epipolarconstraint:Whyisthisuseful?
Reducescorrespondenceproblemto1Dsearchalongconjugate
epipolarlines
AdaptedfromSteveSeitz
Epipolar geometry
EpipolarPlane Baseline
AdaptedfromM.Pollefeys,UNC
Epipoles EpipolarLines
Epipolar geometry:terms
Baseline:linejoiningthecameracenters
Epipole:pointofintersectionofbaselinewiththeimageplane
Epipolarplane:planecontainingbaselineandworldpoint
Epipolar line: intersection of epipolar plane with the image Epipolarline:intersectionofepipolarplanewiththeimage
plane
Allepipolarlinesintersectattheepipole
Anepipolarplaneintersectstheleftandrightimageplanesin
epipolarlines
Grauman
Epipolar constraint
Potentialmatchesforp havetolieonthecorresponding
epipolar line l epipolarlinel .
Potentialmatchesforp havetolieonthecorresponding
epipolarlinel.
Source:M.Pollefeys
http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html
Example
Example:convergingcameras
Aspositionof3dpoint
varies,epipolarlines
rotateaboutthe
baseline
FigurefromHartley&Zisserman
Example:motionparallelwithimageplane
FigurefromHartley&Zisserman
Example:forwardmotion
e e
ee
Epipole has same coordinates in both images Epipolehassamecoordinatesinbothimages.
Pointsmovealonglinesradiatingfrome:Focusofexpansion
FigurefromHartley&Zisserman
Foragivenstereorig,howdoweexpresstheepipolar g g, p p p
constraintsalgebraically?
Stereogeometry,withcalibratedcameras
Iftherigiscalibrated,weknow:
how to rotate and translate camera reference frame 1 to get to howtorotate andtranslate camerareferenceframe1togetto
camerareferenceframe2.
Rotation:3x3matrix;translation:3vector.
Grauman
Rotationmatrix
Express3drotationas p
seriesofrotations
aroundcoordinateaxes
b l
| o
byangles
| o , ,
Overallrotationis
productofthese
elementary rotations: elementaryrotations:
z y x
R R R R =
Grauman
3d rigid transformation 3drigidtransformation

'
T RX X + =
'
Grauman
Stereogeometry,withcalibratedcameras
Cameracenteredcoordinatesystemsarerelatedbyknownrotation
T RX X + =
'
R andtranslationT:
Grauman
Crossproduct
Vector cross product takes two vectors and Vectorcrossproducttakestwovectorsand
returnsathirdvectorthatsperpendicularto
bothinputs.
Sohere,cisperpendiculartobothaandb,
hi h h d d 0 whichmeansthedotproduct=0.
Grauman
Fromgeometrytoalgebra
T RX X' + =
T T RX T X T + =
'

( ) ( ) RX T X X T X
'
=
'

'
0 =
T T RX T X T +
RX T =
0 =
Normaltotheplane
Grauman
Matrixformofcrossproduct p
Canbeexpressedasamatrixmultiplication. Ca be e p essed as a at u t p cat o
Grauman
Fromgeometrytoalgebra
T RX X' + =
T T RX T X T + =
'

( ) ( ) RX T X X T X
'
=
'

'
0 =
T T RX T X T +
RX T =
0 =
Normaltotheplane
Grauman
Essentialmatrix
( ) 0 =
'
RX T X
( ) 0 =
'
RX T X x
Let
R T E x =
0 =
'
EX X
T
0 EX X
Thisholdsfortheraysp andp thatare
parallel to the camera centered position
0 Ep p'
T
E is called the essential matrix which relates corresponding
paralleltothecameracenteredposition
vectorsX andX,sowehave:
0 = Ep p'
E iscalledtheessentialmatrix,whichrelatescorresponding
imagepoints[LonguetHiggins1981]
Grauman
Essentialmatrixandepipolarlines
0 =
'
T
Ep p
Epipolarconstraint:ifweobservepointp in
oneimage,thenitspositionpinsecondimage
i h di i h
p p
mustsatisfythisequation.
Ep
isthecoordinatevectorrepresentingthe
epipolarlineassociatedwithpointp
p E
'
T
isthecoordinatevectorrepresentingthe
epipolarlineassociatedwithpointp
Grauman
Essentialmatrix:properties
Relatesimageofcorrespondingpointsinbothcameras,given g p g p , g
rotationandtranslation
Assumingintrinsicparametersareknown
R T E x =
Grauman
Essentialmatrixexample:parallelcameras
=
T
T
I R
] 0 0 [ d
= =
=
T
]R [T E
T
x
] 0 , 0 , [ d
000
00d
0 d 0 0d 0
0 =
'
T
Ep p p p
Fortheparallelcameras,
imageofanypointmustlieon g y p
samehorizontallineineach
imageplane.
Grauman
image I(x y)
image I(x y)
Disparity map D(x y) imageI(x,y)
imageI (x ,y )
DisparitymapD(x,y)
(x y)=(x+D(x y) y) (x ,y )=(x+D(x,y),y)
Whataboutwhencamerasopticalaxesarenotparallel?
Grauman
Stereo image rectification Stereoimagerectification
Inpractice,itis
convenientifimage
li th i l scanlinesaretheepipolar
lines.
reproject image planes onto a common reprojectimageplanesontoacommon
planeparalleltothelinebetweenopticalcenters
pixelmotionishorizontalafterthistransformation
twohomographies(3x3transforms),oneforeachinput g p ( ), p
imagereprojection
AdaptedfromLiZhang
C.LoopandZ.Zhang.ComputingRectifyingHomographiesforStereoVision.CVPR1999.
Stereo image rectification: example Stereoimagerectification:example
Source:AlyoshaEfros
Stereo reconstruction: main steps Stereoreconstruction:mainsteps
Calibrate cameras Calibratecameras
Rectifyimages
Compute disparity Computedisparity
Estimatedepth
Grauman
Correspondenceproblem
Multiplematch
hypothesessatisfy
epipolar epipolar
constraint,but
whichiscorrect?
FigurefromGee&Cipolla1999
Grauman
Correspondenceproblem
Beyondthehardconstraintofepipolargeometry,thereare
softconstraintstohelpidentifycorrespondingpoints
Similarity
Uniqueness
Ordering
Disparitygradient
To find matches in the image pair we will assume Tofindmatchesintheimagepair,wewillassume
Mostscenepointsvisiblefrombothviews
Imageregionsforthematchesaresimilarinappearance
Grauman
Correspondence problem Correspondenceproblem
Source:AndrewZisserman
Intensityprofiles
Source:AndrewZisserman
Correspondence problem Correspondenceproblem
Neighborhoodofcorrespondingpointsaresimilar
inintensitypatterns.
Source:AndrewZisserman
Normalizedcrosscorrelation
Source:AndrewZisserman
Correlationbasedwindowmatching
Source:AndrewZisserman
Dense correspondence search Densecorrespondencesearch
Foreachepipolarline
Foreachpixel/windowintheleftimage p g
comparewitheverypixel/windowonsameepipolarlineinrightimage
pickpositionwithminimummatchcost(e.g.,SSD,correlation)
AdaptedfromLiZhang
Grauman
Texturelessregions
Texturelessregionsare
Source:AndrewZisserman
nondistinct;high
ambiguityformatches.
Grauman
Effectofwindowsize
Source:AndrewZisserman
Grauman
Effect of window size Effectofwindowsize
W = 3 W = 20
Wantwindowlargeenoughtohavesufficientintensity
variation,yetsmallenoughtocontainonlypixelswithabout
the same disparity
FiguresfromLiZhang
thesamedisparity.
Grauman
Foreshorteningeffects
Source:AndrewZisserman
Grauman
Sparsecorrespondencesearch
Restrict search to sparse set of detected features Restrictsearchtosparsesetofdetectedfeatures
Ratherthanpixelvalues(orlistsofpixelvalues)usefeaturedescriptorand
anassociated featuredistance
Still narrow search further by epipolar geometry Stillnarrowsearchfurtherbyepipolargeometry
Grauman
Uniqueness
Foropaqueobjects,uptoonematchinrightimageforevery
pointinleftimage
FigurefromGee&
Cipolla1999 Grauman
Orderingconstraint
Pointsonsame surface (opaqueobject)willbeinsameorder
in both views inbothviews
FigurefromGee&
Cipolla1999 Grauman
Ordering constraint
Wontalwayshold,e.g.considertransparentobject,oran
occluding surface occludingsurface
FiguresfromForsyth&Ponce Grauman
Disparitygradient
Assumepiecewisecontinuoussurface,sowantdisparity
estimatestobelocallysmooth y
FigurefromGee&
Cipolla1999 Grauman
Scanlinestereo
Trytocoherentlymatchpixelsontheentirescanline
Differentscanlines arestilloptimizedindependently
Left image Right image
i
n
t
e
n
s
i
t
y
Grauman
Shortestpathsforscanlinestereo
Left image
Right image
I
I
'
left
S
Ri ht
q
t

s
i
o
n
Right
occlusion
S
L
e
f
t
o
c
c
l
u
s
t
right
S
p s
Canbeimplementedwithdynamicprogramming
Ohta&Kanade85,Coxetal.96
Slidecredit:Y.Boykov
Coherentstereoon2Dgrid
Scanline stereogeneratesstreakingartifacts
Cantusedynamicprogrammingtofind
spatially coherent disparities/ spatiallycoherentdisparities/
correspondencesona2Dgrid
Grauman
Asenergyminimization
I
1
I
2
D
W
1
(i) W
2
(i+D(i))
D(i)
) ( ) , , (
smooth 2 1 data
D E D I I E E | o + =
( )

D D E ) ( ) (
( )
2

( )

=
j i
j D i D E
, neighbors
smooth
) ( ) (
( )
2
2 1 data
)) ( ( ) (

+ =
i
i D i W i W E
Grauman
Asenergyminimization
I
1
I
2
D
W
1
(i) W
2
(i+D(i))
D(i)
) ( ) (
h 2 1 d
D E D I I E E | o + = ) ( ) , , (
smooth 2 1 data
D E D I I E E | o +
( )

= j D i D E
smooth
) ( ) (
( )
2
2 1 data
)) ( ( ) (

+ = i D i W i W E
Energyfunctionsofthisformcanbeminimized
using graph cuts
( )

j i, neighbors
s oo
( )
2 1 data
)) ( ( ) (

i
usinggraphcuts
Y.Boykov,O.Veksler,andR.Zabih,FastApproximateEnergyMinimizationviaGraph
Cuts,PAMI2001
Source:SteveSeitz
Examples Examples
Grauman
Zkeyingforvirtualreality
Mergesyntheticandrealimagesgivendepthmaps
Kanadeetal.,CMU,1995
Zkeyingforvirtualreality
Kanadeetal.,CMU,1995
http://www.cs.cmu.edu/afs/cs/project/stereomachine/www/zkey.html
Virtualviewpointvideo
C.Zitnicketal,Highqualityvideoviewinterpolationusingalayeredrepresentation,SIGGRAPH
2004.
Virtualviewpointvideo
http://research.microsoft.com/IVM/VVV/
Multibaseline Stereo MultibaselineStereo
Basic Approach BasicApproach
Chooseareferenceview
UseyourfavoritestereoalgorithmBUT y g
replacetwoviewSSDwithSSDoverallbaselines
Limitations
Mustchooseareferenceview
Visibility:selectwhichframestomatch
[Kang,Szeliski,Chai,CVPR01]
Szeliski
Choosing the Baseline ChoosingtheBaseline
Large Baseline Large Baseline Small Baseline Small Baseline
Whatstheoptimalbaseline?
Toosmall:largedeptherror
Szeliski
Toolarge:difficultsearchproblem
Eff t f B li E ti ti EffectofBaselineonEstimation
Szeliski
Szeliski
EpipolarPlane Images [Bolles 87] Epipolar PlaneImages[Bolles87]
http://www.graphics.lcs.mit.edu/~aisaksen/projects/drlf/epi/
Lesson: Lesson: B f B f l i l i
Szeliski
Lesson: Lesson:Beware of Beware of occlusions occlusions
Active stereo with structured light Activestereowithstructuredlight
camera 1 camera 1
Li Zhangs one-shot stereo
projector projector
Projectstructuredlightpatternsontotheobject
camera 2
Szeliski
simplifiesthecorrespondenceproblem
Plane Sweep Stereo PlaneSweepStereo
S f il f l th h l Sweepfamilyofplanesthroughvolume
projective projective re resamplingof( samplingof(X,Y,Z X,Y,Z))
composite composite
inputimage inputimage
each plane defines an image composite homograph
virtualcamera virtualcamera
Szeliski
each plane defines an image composite homography
Plane Sweep Stereo PlaneSweepStereo
Foreachdepthplane p p
computecomposite(mosaic)image mean
computeerrorimage variance p g
converttoconfidenceandaggregatespatially
Selectwinningdepthateachpixel
Szeliski
Plane sweep stereo Planesweepstereo
Reorder(pixel/disparity)evaluationloops e o de (p e / d spa ty) e a uat o oops

foreverypixel, foreverydisparity
foreverydisparity foreverypixel
Szeliski
computecost computecost
framework framework
1 For every disparity compute raw matching 1. Foreverydisparity,computeraw matching
costs
Why use a robust function? Whyusearobustfunction?
occlusions,otheroutliers
Can also use alternative match criteria
Szeliski
Canalsousealternativematchcriteria
framework framework
2. Aggregate costs spatially 2. Aggregatecostsspatially
Here,weareusingaboxfilter
(efficientmovingaverage
implementation)
Canalsouseweightedaverage,
[nonlinear]diffusion
Szeliski
framework framework
3 Choose winning disparity at each pixel 3. Choosewinningdisparityateachpixel
4. Interpolatetosubpixel accuracy
E(d)
d
d*
Szeliski
Traditional Analysis TraditionalAnalysis
Advantages: Advantages:
givesdetailedsurfaceestimates
fastalgorithmsbasedonmovingaverages g g g
subpixeldisparityestimatesandconfidence
Limitations: Limitations:
narrowbaseline noisyestimates
failsintexturelessareas
getsconfusednearocclusionboundaries
Szeliski
Uncalibrated case Uncalibratedcase
Whatifwedontknowthecameraparameters?
Review:Perspectiveprojection
Image
plane
Focal
Optical
length
Camera
frame
p
axis
Scenepoint Imagecoordinates
Thusfar,incamerasreferenceframeonly.
Grauman
Review:Cameraparameters
Extrinsic:locationandorientationofcameraframewith
respect to reference frame
p
respecttoreferenceframe
Intrinsic:howtomappixelcoordinatestoimageplane
coordinates
Reference
frame
Camera 1 Camera1
frame
Grauman
Review:Extrinsiccameraparameters
) ( T P R P =
w c
Worldreference
frame
Camerareference
frame
( )
T
Z Y X = P ( )
c
Z Y X , , = P
Grauman
Review:Cameraparameters
Extrinsic:locationandorientationofcameraframewith
respect to reference frame
p
respecttoreferenceframe
Intrinsic:howtomappixelcoordinatestoimageplane
coordinates
Reference
frame
Camera1
frame
Grauman
Projectionmatrixforperspectiveprojection
fX
fY
Same thing but written in terms of
Frompinholecamera
model
Z
fX
x =
Z
f
y =
Samething,butwrittenintermsof
homogeneouscoordinates
model
Grauman
Review:Intrinsiccameraparameters
Ignoringanygeometricdistortionsfromoptics,wecan
describe them by: describethemby:
x x im
s o x x ) ( =
x x im
) (
s o y y ) ( =
y y im
s o y y ) (
Coordinatesofprojected
pointincamera
reference frame
Coordinatesof
imagepointin
pixel units
Coordinatesof
imagecenterinpixel
units
Effectivesizeofa
pixel(mm)
referenceframe pixelunits units
Grauman
Review:Cameraparameters
Weknowthatintermsofcamerareferenceframe:
) ( T P R P = ) ( T P R P =
w c
and
( )
T
c
Z Y X , , = P
Substitutingpreviouseqns describingintrinsicandextrinsic
parameters,canrelatepixels coordinates toworldpoints:
) ( T P R
) (
) (
) (
3
1
T P R
T P R


=
w
w
x x im
f s o x
R
i
=Rowiofrotat
matrix
) (
) (
) (
2
T P R
T P R


=
w
y y im
f s o y
matrix
) (
3
T P R
w
Grauman
Review:Projectionmatrix
pointincamera
coordinates
Thiscanberewrittenasa
matrixproductusing
homogeneouscoordinates:
wx
im
wy
im
=
M
int
M
ext
X
w
Y
w
Z
g
w
Z
w
1
w wy y
w wx x
im im
im im
/
/
=
=
0 /
x x
o s f
M
T R
T R
T
T

1 13 12 11
r r r
1 0 0
/ 0
y y
o s f
M
int
=
T R
T R
T
T

3 33 32 31
2 23 22 21
r r r
r r r
M
ext
=
Review:Calibratingacamera
Computeintrinsicandextrinsic
parametersusingobservedcameradata p g
Mainidea
Placecalibrationobjectwithknown
geometryinthescene
G d Getcorrespondences
Solveformappingfromscenetoimage:
estimate M=M
int
M
ext
estimateM M
int
M
ext
Grauman
Review:Projectionmatrix
P
w
inhomog.
Thiscanberewrittenas
amatrixproductusing
homogeneous
=
M
int
M
ext
X
w
Y
w
Z
wx
im
wy
im
homogeneous
coordinates:
Z
w
1
M
w
w
im
x
P M
=
1
productM issingle
projectionmatrix
encoding both extrinsic w wx x
i i
/ =
w
w
im
P M
P M

2
3
encodingbothextrinsic
andintrinsic
parameters
w wy y
w wx x
i i
im im
/
/
=
w
w
im
y
P M
=
3
2
LetM
i
berowi
ofmatrixM
w wy y
im im
/
Grauman
Review:Estimatingtheprojectionmatrix
Foragivenfeaturepoint
w
im
x
P M
=
1
w im
x P M M = ) ( 0
3 1
g p
w
w
im
P M
P M

2
3
w im
) (
3 1
) (
w
w
im
y
P M
=
3
2
w im
y P M M = ) ( 0
3 2
Grauman
Review:Estimatingtheprojectionmatrix
w im
x P M M = ) ( 0
3 1
w im
y P M M = ) ( 0
3 2
(
X
Expandingthisfirstequation,we
have:
| | | | 0 * 34 33 32 31 14 13 12 11 =
(
(
(
(

w
w
im
Y
X
m m m m x m m m m | | | | 0
1
34 33 32 31 14 13 12 11
(
(

w
im
Z
m m m m x m m m m
( ) ( ) 32 12 31 11 m x Y m Y m x X m X im w w im w w + ( ) ( )... 32 12 31 11 m x Y m Y m x X m X im w w im w w +
( ) ( ) 0 ... 34 14 33 13 = + + m x m m x Z m Z im im w w ( ) ( ) 0 ... 34 14 33 13 + + m x m m x Z m Z im im w w
Grauman
Review:Estimatingtheprojectionmatrix
w im
x P M M = ) ( 0
3 1
w im
y P M M = ) ( 0
3 2
im w im w im w im w w w
im w im w im w im w w w
y Z y Y y X y Z Y X
x Z x Y x X x Z Y X


1 0 0 0 0
0 0 0 0 1
0
0
=
Grauman
Review:Estimatingtheprojectionmatrix
Thisistrueforeveryfeaturepoint,sowecanstackupn
observedimagefeaturesandtheirassociated3dpointsin
single equation:
Pm = 0
singleequation:
) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 (
0 0 0 0 1 x Z x Y x X x Z Y X
P
m
Pm=0
) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 ( ) 1 (
1 0 0 0 0
im w im w im w w w w
im w im w im w im w w w
y Z y Y y X y Z Y X
im


0
0
=
) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) (
) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) (
1 0 0 0 0
0 0 0 0 1
n n n n n n n
im
n n n
n n n n n n n n n n
im w im w im w w w w
im w im w im w im w w w
y Z y Y y X y Z Y X
x Z x Y x X x Z Y X



0
0
=
Solve for m
ij
s (the calibration information) [F&P
0
Solveform
ij
s (thecalibrationinformation)[F&P
Section3.1]
Grauman
Summary:cameracalibration
Associateimagepointswithscenepointsonobjectwith
kknowngeometry
Usetogetherwithperspectiveprojectionrelationshipto
estimate projection matrix estimateprojectionmatrix
(Canalsosolveforexplicitparametersthemselves)
Grauman
Whenwouldwecalibratethisway?
Makessensewhengeometryofsystemisnotgoingtochange g y y g g g
overtime
Whenwoulditchange?
Grauman
Weakcalibration
Wanttoestimateworldgeometrywithoutrequiring
calibrated cameras calibratedcameras
Archivalvideos
Photosfrommultipleunrelatedusers
Dynamic camera system Dynamiccamerasystem
Mainidea:
Estimateepipolar geometryfroma(redundant)setof
pointcorrespondencesbetweentwouncalibrated cameras
Grauman
Uncalibratedcase
p M p
int
=
Foragiven
camera:
Camera
coordinates
p p
int
camera:
coordinates
1
p M p

So,fortwocameras(leftandright):
) ( ) (
int ,
left left
left
p M p =
1
p M p

=
Cameracoordinates
Imagepixel
coordinates
) ( int ,
) (
right right
right
p M p =
Internalcalibration
matrices,onepercamera
Grauman
Uncalibratedcase:
fundamentalmatrix
) ( ) (
1
int ,
left left
left
p M p

=
1
p M p

=
0
) ( ) (
=
T
left right
Ep p
Frombefore,the
essential matrix E
) ( int ,
) (
right right
right
p M p =
) ( ) ( f g
essential matrixE.
( ) ( ) 0
1 1
T

M E M ( ) ( ) 0
1
int ,
1
int ,
=

left left right right
p M E p M
( ) ( ) 0
1
int , int ,
=
T T
left left right right
p EM M p
0 =
T
left right
p F p
f g
Fundamentalmatrix
Grauman
Fundamentalmatrix
Relatespixelcoordinatesinthetwoviews
Moregeneralformthanessentialmatrix:weremoveneedto
knowintrinsicparameters
If ti t f d t l t i f d i Ifweestimatefundamentalmatrixfromcorrespondencesin
pixelcoordinates,canreconstructepipolargeometrywithout
intrinsicorextrinsicparameters
Grauman
ComputingFfromcorrespondences
( )
1
int . int ,
T
=
left right
EM M F
0 =
T
l ft i ht
p F p
Camerasareuncalibrated:wedontknowE orleftorright
0
left right
p F p
M
int
matrices
EstimateFfrom8+pointcorrespondences.
Grauman
ComputingFfromcorrespondences
0 =
T
left right
p F p
Eachpoint
correspondence
t
left ight
generatesone
constraintonF
Collect n of these Collectnofthese
constraints
Solveforf,vectorofparameters.
Grauman
Stereopipelinewithweakcalibration
So,wheretostartwithuncalibrated cameras?
NeedtofindfundamentalmatrixFand thecorrespondences(pairsofpoints
(u v) (u v)) (u,v )(u,v)).
1)Findinterestpointsinimage(moreonthislater)
2)Computecorrespondences
3) Compute epipolar geometry 3)Computeepipolar geometry
4)Refine
ExamplefromAndrewZisserman
Stereopipelinewithweakcalibration
1)Findinterestpoints
Grauman
Stereopipelinewithweakcalibration
2)Matchpointsonlyusingproximity
Grauman
Putativematchesbasedoncorrelation
h search
Grauman
RANSACforrobustestimationofthe
f d l i fundamentalmatrix
Select random sample of correspondences Selectrandomsampleofcorrespondences
ComputeFusingthem
This determines epipolar constraint Thisdeterminesepipolarconstraint
Evaluateamountofsupport inlierswithinthresholddistance
ofepipolarline p p
Choose F with most support (inliers) ChooseFwithmostsupport(inliers)
Grauman
Putativematchesbasedoncorrelation
h search
Grauman
Prunedmatches
Correspondencesconsistentwithepipolargeometry
Grauman
Resultingepipolargeometry g p p g y
Grauman
Slide Credits SlideCredits
Kristen Grauman for most KristenGrauman formost,
RickSzeliski andothersasnoted
Today: Stereo Today:Stereo
Human stereopsis & stereograms Humanstereopsis &stereograms
Epipolar geometryandtheepipolar constraint
Caseexamplewithparallelopticalaxes
Generalcasewithcalibratedcameras
Correspondencesearch
TheEssentialandtheFundamentalMatrix e sse t a a d t e u da e ta at
Multiviewstereo
Roadmap
Previous: Image formation filtering local features (Texture) Previous:Imageformation,filtering,localfeatures,(Texture)
Tues:FeaturebasedAlignment
Stitchingimagestogether
Homographies,RANSAC,Warping,Blending
Globalalignmentofplanarmodels
Today: Dense Motion Models Today:DenseMotionModels
Localmotion/featuredisplacement
Parametricopticflow
Noclassesnextweek:ICCVconference
Oct6
th
:Stereo/Multiview:Estimatingdepthwithknown
i t intercamerapose
Oct8
th
:Structurefrommotion:Estimationofposeand3D
structure
Factorizationapproaches
Globalalignmentwith3Dpointmodels