Image Fusion and

Restoration using
Inpainting
E9:241 DIP TERM Submitted by:
Sourav Mishra
PROJECT Mtech, CDS
Date – 26/11/2018
06-02-01-10-51-18-1-15700
Contents
●
Image Restoration using Inpainting
- Variational and PDE based Inpainting and brief description
- PDE based Inpainting – with and without external
information
- Results and Discussion
●
Image Fusion using Blending Techniques
- Simple Alpha Blending and Feathering
- Pyramid Based Blending
- Results and Discussion
 Code and Data Files
●
Inpainting part:
Image.py – class which returns an object containing the original,
corrupted versions of the image, the mask and the normalized and
unnormalized gradients of the image.
Save_Dataset.py – takes the flenames of original, corrupted and mask
fles – makes the image objects and dumps them into a fle Datasets.pkl
InPainting.py – Implements the two inpainting algorithms.
InpaintingMain.py – The code which calls InPainting.py on images and
returns the results into a destination folder.
Data is organized into 3 folders – Original, Corrupted and Mask.
Additionally, results are written into Results folder. All of these are inside
a Datasets folder.
FileNames.txt – Contains names of images which are to be read and
saved by Save_Dataset.py
Code and Data Files
●
Blending Part -
All data is organized into a folder named ‘Datasets/Blending’.
ImageBlender.py – Implements pyramid blending and
feathering.
BlendingMain.py – Calls ImageBlender.py to get the
blending done.
●
For Inpainting: Organize dataset – call Save_Dataset.py – run
InpaintingMain.py with Datasets.pkl fle loaded.
●
For Blending: Organize dataset – run BlendingMain.py
File Naming Convention
●
Inpainting: for each image to be inpainted there
have to be 3 fles in corresponding folders – name-
original, name-corrupted and name-mask.
●
The Image class supports only grayscale images.
So in order to make a color image object – make 3
objects – one for each channel and inpaint each
channel separately.
●
Blending: No naming convention as such.
●
Modify the folder paths in the codes appropriately.
Variational and PDE Approach
●
Variational Based Approach: fnd an image u which
minimizes a functional

●
Variational Based Approaches are implemented
through PDEs. Solution image is evolved through
time as
Approaches Considered
●
Pure PDE based approach due to Beltramio and
Sapiro, motivated by museum artists:

●
Propagate external information (here smoothness)
along the edges into the damaged region. Edges
are preserved.
●
Anisotropic difusion is done after every few steps.
Approaches Considered
●
Based on Curvature fow. Assuming that the
normalized gradients in the damaged region are
supplied.

●
The previous approach used only local
information.
●
Variational Approach can be converted to PDE via
Euler Lagrange Equations.
●
Image evolves till the derivative vanishes.
Inpainting Steps
●
The mask is dilated and borders are blurred. The
same is also done for the corrupted image in order
to avoid high magnitude of gradients at the
transition. This is achieved through a few
iterations of Laplace equation. This could also be
done by a low pass fltering operation near the
damaged regions.
●
Do anisotropic difusion after every few steps of
inpainting. This is like bleeding colors from the
neighbouring good pixels to paint the damaged
regions.
Results and Discussion
●
Images are shown in the following order starting
from top left in anti clockwise direction:
corrupted – mask – PDE inpainted – curvature fow
inpainted.
●
Inpainting can be used for removal of objects and
texts.
●
Other applications include restoration of damaged
art works, color restoration and compressed
sensing. Color images are restored by applying
the algorithm on each channel separately.
Turtle
Horse
Jordis
Lena
Barbara
MontyPython – Color Image
Discussion
●
Both methods are good in inpainting small and fat
regions. PDE method needs more iterations to fll
large regions.
●
Curvature fow method is faster and repairs most
of the regions than PDE method – fewer
computations as normalized gradients are
supplied. Noticeable diference in runtimes.
●
Texture is not preserved – Barbara image. For this
purpose use Simultaneous Structure and Texture
Inpainting.
Discussion
●
Both methods need a mask which need not always
be available – so using dictionary methods can be
advantageous.
●
Resulting images appear darker due to presence
of few steps of anisotropic difusion.
●
Both algorithms are prone to numerical
instabilities – few white pixels in places close to
the inpainted regions of the Turtle image. So
parameter tuning is a very tedious process.
Parameters used
●
Turtle and Horse – step size = 0.05, inpainting
iterations = 2000, anisotropic difusion iterations
= 100
●
Other Images – step size = 0.05, inpainting
iterations = 3000, anisotropic difusion iterations
= 100
●
Running time is on average 10-20 min for PDE
based approach and 5-10 min for curvature fow
approach.
Modifcations Proposed
●
Inpainting algorithms can be numerically unstable and
large number of iterations may not always be feasible.
For small number of iterations, these modifcations may
work:
1. after inpainting, if there are small white pixels
remaining which resulted due to inpainting, then erode
them. Alternatively – set them to be black and contrast
stretch and inpaint again with the same mask.
2. fnally apply contrast stretching to improve the
appearance of the resulting image as it looks darker
due to difusion iterations.
Image Fusion
●
Image fusion is the blending of two images.
●
Requires a source, a target and a mask.
●
The images must be perfectly aligned with each
other.
●
Approaches considered – simple cut and paste
with feathering and pyramid based fusion.
●
Finds applications in compression, denoising,
texture synthesis and panorama creation.
Image Fusion steps
●
First align the source and target images perfectly and
create the masks according to the regions to be
blended.
●
Simple cut and paste: mask*source + (1-mask)*target
●
Feathering – Blur the mask using a Gaussian low pass
flter of specifed sd.
●
Pyramid blending – Construct Laplacian pyramids of
source and target and Gaussian pyramid of mask. At
each stage do feathering. Finally, collapse the
resulting pyramid.
Results and Discussion
●
Images are shown in the order left – right as:
alpha = 0, 5, 10, 20, 40, 80.
●
A gaussian mask of same sd as alpha is used to
blur the mask before multiplying with the source
and target. Other choices could be a distance
transform.
●
For Pyramid blending, the order is levels: 1, 2, 3,
4, 5, 6 from left to right.
●
Before blending, alignment is ensured.
Apple and Orange - feathering
Apple Gaussian Pyramid
Apple Laplacian Pyramid
Orange Gaussian Pyramid
Orange Laplacian Pyramid
Pyramid of Final Image
Apple and Orange - Pyramid
Another Example - Pyramids
Face transfer
Face transfer - feathering
Discussion
●
With increase in alpha (sd of gaussian smoother),
the mask gets bigger and neighbouring target
pixels get some weight – so frst the color
separation between the images disappears and
then as the whole mask is smoothed out (size of
gaussian is bigger than the mask size), the source
part vanishes.
●
At a given spatial location, feathering weighs all
frequencies equally – this could lead to aliasing.
●
Pyramid method mitigates this issue.
Face transfer - Pyramids
Discussion
●
With increase in the number of levels being
combined, the colors of the source and target
start becoming similar, although the edge
between them never vanishes.
●
The condition of equality of pixel intensity and
gradient at the boundaries was not explicitly
imposed which is why the color mismatch is
observed.
●
This is taken care of by Poisson blending with
gradient mixing.
Comparison
●
As seen from the images, pyramid blending seems
better than simple cut and paste and feathering.
●
The idea behind pyramid blending is to weight the
contribution of diferent frequency contents diferently.
It is difcult to separate these from the ft directly, so
Laplacians are used instead.
●
From the pyramid collapse step, it is clear that the high
frequency contents are weighted higher than others.
●
Transition regions are undetectable but textures might
be diferent depending on the images being combined.
Codes and fles
●
Inpainting codes – InpaintingMain.py,
Inpainting.py, Image.py, Save_Dataset.py
●
Inpainting fles – Datasets.pkl (all combined),
(folders)Corrupted, Mask, Original, Results,
FileNames.txt
●
Blending codes – ImageBlender.py,
BlendingMain.py
●
Blending fles – Blending folder – data and results
●
Corrupted images, masks and one set for blending
were generated using opencv manually.
References
●
Processing of fat and non-fat image information on
arbitrary manifolds using Partial Diferential
Equations – PhD thesis, Beltramio, pg. 30-36
●
Fourth-Order PDEs for Image Restoration, talk at
DAMTP, University of Cambridge, London by Carola-
Bibiane Schonlieb
●
Slides from 15-643: Computational Photography,
Alexei Efros, CMU, Spring 2010
●
For parameter choice and anisotropic difusion –
MATLAB/Python codes for the Inpainting Problem –
github repository – Carola-Bibiane Schonlieb
Thank You!