3D Reconstruction USING MULTIPLE 2D IMAGES

Kifayat ullah
Cecos university of IT and emerging sciences and engineering Peshawar hayat abad

 Stage 1. The Total Caleb system proposed by Bougnoux et

Abstract:
Multi software used to get the results of multiple images.
The (SFM) and (CMVS) used to obtain the sparse and dense
reconstruction of sequential images collections. By taken
images of engine through robot from different view to
quantify the quality of (SFM) data in relation. We have
implemented a number of comparison b/w different parts of
engine in order to assess the mesh deviation and the
reconstruction accuracy. The (SFM) and (CMVS) techniques
and methods were mostly questioned by creating a complete
3D digital replication of engine.
1. INTRODUCTION
T HE 3D reconstruction is that type of process through
which we can achieve the real and authentic shape of an
object. The process can be achieved either active or
passive methods. There is an increasing need for geometric
3D models in the movie industry, the games industry,
reverse engineering mapping (Street View) and others.
Generating these models from a sequence of images is much
cheaper than previous techniques (e.g. 3D scanners). These
techniques have also been able to take advantage of the
developments in digital cameras and the increasing
Resolution and quality of images they produce along with
the large collections of imagery that have been established
on the Internet (for example, on Flickr).
al. can complete the image match [4]. The camera
calibration and 3D reconstruction are semi-automatic.
Stage 2. A 3D surface automatic building system was put
forward by Pollefeys at K. U. Leuven University in-
Belgium. The System adopts the camera self-calibration
technique with variable parameters [5]. The System requires
users to use hand-held video camera to screen a series of
images about the object and to match the corresponding
points of the images to achieve self-calibration and layered
reconstruction, but the gathered object images must be
comprehensive.
Stage 3. A similar 3D reconstruction system was developed
by computer vision study team of Cambridge University [6].
The system can calibrate intrinsic parameters through
manual designating vanishing points which are formed in the
image by the three groups of spatial orthogonal parallel
lines. But its applicability and automaticity are not enough.
Stage 4. A fully automatic 3D reconstruction method based
on images without manual modeling [7]. This method
extracts 3D information from 2D images directly. First
Feature points are extracted and then they are matched
together. To calculate coordinates of 3D points, fundamental
matrix with extrinsic and intrinsic parameters of camera is
used.
3. METHODOLOGY
Our 3D Reconstruction Pipeline consists of the following
steps:

Extract Sift key

Calculate Intrinsic points and its
Input Images
Camera Parameter matches in
consecutive frames

Calculate [R|T]
Make graph of Extract Fundamental
parameter for all
The objective of this report is to identify the various matched 3d points of
frames
frames
and Essential matrix

approaches to generating sparse 3D reconstructions using the

Structure from Motion (SFM) algorithms and the methods to
generate dense 3D reconstructions using the Multi View
Merge graph and
Stereo (MVS) algorithms. Generate Sparse Generate dense
Calculate image
points for all dense
points points points
2. LITERATURE SURVEY
3D navigation, virtual reality and so on. At present, some
outstanding 3D reconstruction methods appear in
international community as follows. Reconstruction is Coloring of Dense
points
widely applied in robot

3.1 Technical Approach Given the complexity
involved in creating a full scale SFM and MVS
implementation from scratch, the approach taken on this
project was to implement the Structure from Motion
algorithms by engine on top of the material covered in class
and sample code found online. These results were compared
with those produced by the open source packages.
3.2 Sorting the Photo Collection
One of the first steps involved when dealing with an
unordered photo collection is to organize the available
images such that image are grouped into similar views.
The data consist of 200 images and then matching and
reconstruction take a total of 3 hours on a cluster with 213
compute cores. For this project, the SIFT algorithm was used
to compare the images in the collection and images with a
high number of correspondences were considered to be
/close together and therefore good candidates for SFM
process..
3.3 Feature Detection and Matching
Given a feature in an image, what is the corresponding
projection of the same 3D feature in the other image? This is
an ill-posed problem and therefore in most cases very hard
to solve. It is not clear that not all possible image features
are good for matching. Often Points are used. Many interest
point detector exist. In Schmidt ET. Al. concluded that the
Harris corner detector gives the best results. Later more
robust techniques developed, one of them is SIFT which is
used in our project.
In the Photo Engine project, the approach used for feature
detection and matching was to:
 Find feature points in each image using SIFT.
 for each pair of images match key points using the
approximate nearest neighbors, estimate the
fundamental matrix for the pair using RANSAC
(use 8 point algorithm followed by non-linear
tenement) and remove matches that are outliers to
the re-covered fundamental matrix. If less than 20
matches remain, then the pair was considered not
good.
 Organize the matches into tracks, where a track is a
connected set of matching key points across
multiple images.
3.4 Structure from Motion (SFM)
In the Photo of engine project, the approach used for the 3D
reconstruction was to recover a set of camera parameters and
a 3D location for each track. The recovered parameters
Should be consistent, in that the projection error is
minimized (a nonlinear least squares problem that was
solved using Liebenberg Marquardt algorithm) Rather than
estimate the parameters for all cameras and tracks at once,
they took an incremental approach, adding one camera at a
time.
The first step was to estimate the parameters for a single pair
of images. The initial pair should have a large number of
feature matches, but also a large baseline, so that the 3D
Locations of the observed points are well-conditioned then,
another image was selected that observes the largest number
of tracks whose 3D locations have already been estimated. A
New camera's extrinsic parameters are initialized using the
DLT (direct linear transform)
Technique inside a RANSAC procedure. DLT also gives an
estimate of K, the intrinsic Camera parameter matrix. Using
the estimate from K and the focal length estimated from
The EXIF tags of the image, a reasonable estimate for the
focal length of the new camera can be computed.
The next step is to add the tracks observed by the new
camera into the optimization. A track is added if it is
observed by at least one other camera and if triangulating the
track gives a well-conditioned estimate of its location and
parts. This procedure is repeated, one image at a time until
no remaining image observes any the reconstructed 3D
points. To minimize the objective function at each iteration,
they used the Sparse Bundle Adjustment library.
The run times for this process were a few hours (engine-120
photos) to 3 days.
3.4.1 SFM using Two Images
Structure from Motion techniques using a pair of images
were covered in class. In particular, Estimation of the
fundamental matrix F from point correspondences and
solving the affine Structure from Motion problem using the
Factorization Method proposed by Tomasi and Kanade were
implemented in problem set.
The general technique for solving the structure from motion
problem is to
-estimate the m 2x4 projection matrices Mi (motion) and the
n 3D positions Pij (Structure) from the mxn 2D
correspondences pij (in the affine case, only allow
For translation and rotation between the cameras)
-This gives 2mxn equations in 8m+3n unknowns that can be
solved using the algebraic method or the factorization
method.
-Convert from perspective to metric via self-calibration and
apply bundle adjustment.
For this project, two approaches were investigated for the
scenario where the camera matrices are known (calibrated
cameras):
The first approach is based on the material given:
 Compute the essential matrix E using RANSAC
 Compute the camera matrices P
 Compute the 3D locations using triangulation. This
produces 4 possible solutions of which we select
the one that results in reconstructed 3D points in
front of both cameras.
 Run Bundle Adjustment to minimize the projection
errors by optimizing the position of the 3D points
and the camera parameters.
The second approach utilizes Open CV and is based on the
material given in
 Compute fundamental matrix using RANSAC
(Open CV: find Fundamental Mat ())
 Compute essential matrix from fundamental matrix
and K (HZ 9.12/9.13) Open CV: Compute E =
K.T*F*K
  Decompose E using SVD to get the second camera SFM & Mesh Lab) was used. This dataset was a good choice
matrix P2 (HZ 9.19) (first camera matrix P1 is for testing with because a complete set of images was
assumed at origin - no rotation or translation) available (24 images at approx. 5 deg increments), the
 Compute 3D points using triangulation (Open CV: camera matrices are available, ground truth is available and
no function for triangulation, code your own) a dense 3D model available for comparison.
When dealing with the situation where the intrinsic camera 7. RESULTS
parameters are unknown, one can run the Self Calibration
(also known as Auto Calibration) process to estimate the
camera parameters from the image features. Possible
techniques for Self Calibration include using the single-view
metrology constraints, the direct approach using the Kruppa
equations, the algebraic approach or the stratified approach
3.4.2 SFM using Multiple Images.
With two images, we can reconstruct up to a scale factor.
However, this scale factor will be Different for each pair of
images. How can we find a common scale so that multiple
images can be combined? One approach is to use the
Iterative Closest Point (ICP) algorithm, where we triangulate
More points and see how they fit into our existing scene
geometry. A second approach (and the one used on this
project) is to use the Perspective N-Point (PnP) algorithm
(Also known as camera pose estimation) where we try to
solve for the position of a new camera using the scene points
we have already found. Open CV provides the solve PnP()
And solve PnP Ransac () functions that implement this
technique.
3.5 Multi View Stereo 8. Statistical analysis:
The Multi View Stereo algorithms are used to generate a
dense 3D reconstruction of the object or scene. The
techniques are usually based on the measurement of a DENSE SPARSE POINT
consistency function, a function to measure whether \this 3D Images Time(min)
model is consistent with the input images"Generally, the 82 62.70
answer is not simple due to the effects of noise and
calibration errors. 120 32.46
3.5.1 Color the dense point 200 277.5
Do the cameras see the same color? This approach is valid
for Lambert Ian surfaces only and is based on a
measurement of color variance. images vs time
After finding the 3D point, we need to color those points. 300
Coloring of these points is done in two steps:-
Firstly, for each 3D point we find the corresponding point in 250
the images by using 𝒙=𝑷𝑿 200
time

Set the color of 3D point is equal to color of the 150

corresponding point in Image. images
100
3.5.2 Texture: time
The texture around the points the same. This approach can 50
handle glossy materials, but has problems with shiny 0
objects. It is based on a based on a measurement of 1 2 3
correlation between pixel Patches. One of the following two images
approaches are generally used to build the dense 3D model:
build up the model from the good points. Requires many
views otherwise holes appear. Remove the bad points (start
from the bounding volume and carve away inconsistent Vertices used in images
points). Requires texture information to get a good Images Image sizes vertices
geometry.
82 3376×2704 249450
6. Experiment:
For the First set of tests, the engine dataset, a set of 24 120 3376×2704 266366
images from the Dense MultiView Stereo datasets (visual 200 3376×2704 291750
 Proc. Compute Vision and Pattern Recognition, Demo
images vs vertices Session, 1997.
250 300000 [5] Polleleys M., “Self-calibration and metric 3D
290000 reconstruction from uncalibrated image sequences”,
200 Ph.D. Thesis, Katholieke Universiteit Leuven,
280000
270000 Heverlee, 1999.
vertices

150
260000 [6] Cipolla R., Robertson D. P. and Boyer E.G.,
100 250000 “Photobuilder-3Dmodels of architectural scenes from
240000 uncalibrated images”. Proc. IEEE International
50 Conference on Multimedia Computing and Systems,
230000
0 220000 Firenze, volume 1, pp.25-31, June, 1999.
1 2 3
[7] Jiang Ze-tao, Zheng Bi-na, Wu Min, Wu Wen-
huan, “A Fully Autmatic 3D Reconstruction Method
images
Based on Images”, Computer Science and Information
Engineering, 2009 WRI World Congress, volume 5,
9. Conclusion: pp. 327-331, March-April, 2009.
From above statistical analysis data we concluded that by [8] YUAN Si-cong; LIU Jin-song, “Research on image
increasing number of images in SFM software it will matching method in binocular stereo vision. Computer
increase time for dense sparse point and also increase
Engineering and Applications”, 2008，44 (8): pp. 75-
vertices of images. The Structure from Motion and Multi
View Stereo algorithms provide viable methods for
77.
Engine 3D models of vehicle. The key issues with the [9] N. Snavely, S. Seitz, R. Szeliski. Photo Tourism:
algorithms are they are fairly CPU and memory intensive, Exploring Photo Collections in 3D. ACM
especially when trying to do reconstruction at large scale as Transactions on Graphics (2006)
was done in the Photo engine project. The Visual SFM [10] Y. Furukawa, B Curless, S Seitz, R Szeliski.
implementation was found to be much faster than the other Towards Internet-scale Multi-view Stereo. In
toolkits (a few seconds per image vs. many seconds to a few CVPR (2010)
minutes per image for others). This was mainly due to use of =============Remark================
the GPU for SIFT and the implementation of a multicore
bundle adjustment algorithm.
5. FUTURE WORK.
A lot of Scope is present for Improvement.
1. Removing performance bottleneck.
2. Improving Feature Detection.
3. Creating Mesh from Point Cloud.
4. Applying Texture Mapping.
5. on the Fly Calculation of Intrinsic Camera Matrix.
6. Incorporating Improvements to Handle Radial Distortion
and Skewers.
6. REFERENCES
[1] C. Tomasi and T. Kanade. Shape and motion from
image streams under orthography: A
factorization method. IJCV, 9(2):137-154, November
1992.
[2] M. Pollefeyes, R. Koch, M. Vergauwen and L.
Van. “Automated reconstruction of 3D scenes from
sequences of images”, ISPRS journal of
photogrammetry & remote sensing 55, 251-267, 2000.
[3] Lowe, David G. (1999). "Object recognition from
local scale-invariant features".Proceedings of the
International Conference on Computer Vision 2.
pp.1150–1157.
[4] Bougnoux S. and Robert L., “A fast and reliable
system for off-line calibration of images sequences”,