May 25, 2014

Assoc. Prof. Dr. Maria TROCAN

Associate Professor
Institut Suprieur d'Electronique de Paris
Signal & Image Processing Dept.


Dear Assoc. Prof. Dr. Maria TROCAN,
I wish to submit a training final report entitled Image Super Resolution Techniques
for your review. This report summarizes our research during April - June 2014, in order
to fulfill my knowledge and experience in the field of digital image processing.
Thank you in advance for your kind consideration of the enclosed report.
Sincerely,
Nutasn Ratchano
Enclosure
























"

Summer Training Final Report


Image Super Resolution Techniques


Submitted to

Dr. Maria TROCAN
Associate Professor
Institut Suprieur d'Electronique de Paris
Signal & Image Processing Dept.

16 May 2014


by
Nutasn Ratchano
Electronics and Communication Program
Sirindhorn International Institute of Technology
Thammasat University






#

Abstract


Super Resolution has been one of the main topics of research in Image Processing
in the last twenty years. We should begin with the use of the term super resolution(SR).
Super Resolution algorithms reconstruct a high resolution(HR) image from a set of low
resolution (LR) images of a scene.
Super Resolution techniques are being applied to a variety of fields, such as
obtaining improved still images from video sequence, astronomy and medical imaging.
Increasing Resolution of an image can be done by increase sensor quality but it
increases production cost. Here is where Super Resolution algorithms can be applied,
with less cost and still be able to obtain high resolution image is the main purpose of SR.
Methods of Super resolution algorithms will be reviewed in this report. Each kind
of algorithm has its own advantages. SR performances depend on the application. We
will also propose Optical-Flow based Super Resolution in the end of this report.


















$

Table of Contents


List of Figuresiv
List of Tables ..................................................................................................................... 3
Introduction ....................................................................................................................... 4
1.1 Introduction ......................................................................................................................... 4
1.2 Classification of Super Resolution Algorithms .................................................................. 6
1.3 Problem Description ............................................................................................................. 7
1.3.1 Common Image Acquisition System ................................................................................. 8
1.3.2 Observation Model ............................................................................................................ 9
Motion Estimation .......................................................................................................... 13
2.1 Preliminaries ..................................................................................................................... 14
Reconstruction ................................................................................................................. 22
3.1 Optical-flow based super resolution reconstruction ............................................................. 22
Result ................................................................................................................................ 25
Bibliography: ................................................................................................................... 31
MATLAB code .................................................................................................................. 1













%


List of Figures


Figure 1 : Comparison of visual quality for bilinear interpolation and super resolution 1
Figure 2: LR images and the HR image after applying SR algorithm 2
Figure 3: The basic idea for super resolution 2
Figure 4: Classifications of the Super Resolution algorithms 3
Figure 5: Basic premise of super resolution 4
Figure 6: Common Image Acquisition System 5
Figure 7: Common Image Acquisition System 5
Figure 8: Observation Model relating LR images to HR images 6
Figure 9: Common Super Resolution stages 6
Figure 10: Overview of common super resolution algorithms 10
Figure 11: Procedure to obtain a LR image sequence from a HR image sequence 13
Figure 12: Block-matching frame division in blocks of size m x n. 14
Figure 13: Coarse-to-fine optical flow estimation 16
Figure 14: Super resolution with mis-registration 18
Figure 15: Combination of LR images to create the HR image. 19
Figure 16: Deblurring block. 20


List of Tables
Table 1: PSNR results of super resolution methods..26













&

Chapter 1
Introduction

1.1 Introduction

The term super resolution(SR) is used to describe the process of obtaining an high
resolution image or a sequence of HR images from a set of low resolution observation.
In most electronic imaging applications, images with high resolution are desired
and often required. HR means that pixel density within an image is high, and therefore an
HR image can be offer more details that may be critical in various applications. For
instance, HR medical images are very helpful for a doctor to make a correct diagnosis. It
may be easy to distinguish an object from similar ones using HR satellite images, and the
performance of pattern recognition in computer vision can be improved if an HR image is
provided.
The most direct solution to increase spatial resolution is to reduce the pixel size
(i.e., increase the number of pixel per unit area) by sensor manufacturing techniques. As
the pixel size decreases, however, the amount of light available also decreases. It
generates shot noise that degrades the image quality. Therefore, there exists the limitation
of the pixel size reduction. Another approach for enhancing the spatial resolution is to
increase the chip size but it has many disadvantages. Increasing image sensors quality by
manufacturing is one way to solve this problem but it is not the solution for us, either
because of cost or hardware physical limits, super resolution algorithms are an
inexpensive alternative.


Figure 1: Comparison of visual quality for bilinear interpolation(left) and one
of super resolution algorithms



'







Super resolution is a computationally intensive process. For example some video
applications may require super resolution to be done fast or data must be processed as
they are received. Even it is a trade-off but speed should not be gained at the cost of
robustness. SR algorithms must be robust to sources of image degradations. Including
unknown sensor noise, unknown camera characteristics from frame-to-frame, etc.
Moreover, the algorithms must be driven only by the sensor data as no information about
noise or camera characteristics may be given.
The algorithm proposed by us will be evaluated by both the qualitative result and
quantitative result in the end of the report as well.




Figure 2: LR images and the HR
image after applying SR algorithm
Figure 3: The basic idea for super resolution reconstruction from multiple LR frames. Subpixel motion
provides the complementary information among the LR frames that makes SR reconstruction possible.


(

1.2 Classification of Super Resolution Algorithms
Super resolution was first proposed by Tsai and Huang [2], who suggested a
frequency domain approach to SR. This first approach assumes a translational movement
among the LR images and is based on Fourier shifting properties, aliasing and the
assumption that the HR image is band-limited. In early works, frequency-based methods
were the chosen approaches despite being restricted in the image observation model and
the real scenarios, which they can handle. Now most super resolution algorithms are
working on spatial-based, for its flexibility to model all kinds of image degradations.

Figure 4: Classifications of the Super Resolution algorithms
A. Single-image and multi-frame super-resolution
Super-resolution (SR) aims to produce a high-resolution (HR) image using one or
several observed low-resolution (LR) images by upsampling, deblurring and denoising.
For multi-frame SR, there can involve registration and fusion processes. It is interesting
to point out that some SR algorithms do not involve the denoising process, or some
interpolation algorithms are also referred to as super- resolution algorithms. Basically, the
super- resolution methods can be classified into single-image super- resolution (only one
LR image is observed) and multi-frame super-resolution. The multi-frame super-
resolution for a video sequence can moreover use a recursive estimation approach of
video frames, which is called the dynamic super-resolution. Dynamic super-resolution
makes use of the previously reconstructed HR frame to estimate the current HR frame.
B. Reconstruction-based and learning-based super- resolution
There are two major categories of SR algorithms: they are the reconstruction-
based and learning-based algorithms. Reconstruction-based algorithms use the data
fidelity function, usually with the prior knowledge to regularize the ill-posed solution.
Gradients are the main features to be used as the prior knowledge. Some more general
prior knowledge using nonlocal self-similarity was also proposed. Learning-based
algorithms moreover utilize the information from training images. Generally, when a LR
image is observed, a patch within that LR image is extracted and searched within the


)

trained dictionary for estimating a suitable HR patch that reconstructs the HR image.
Some recent investigations make use of the sparse representation for estimating the HR
pair and training the dictionary. By combining the use of prior knowledge and dictionary
training, some unified frameworks for single-image SR were proposed to further
regularize the estimated solution
1.3 Problem Description
How can we obtain an HR image from multiple LR images? The basic idea for
increasing the spatial resolution in SR techniques is the availability of LR images
captured from the same scene. Typically in super resolution the LR images represent
different perspectives at the same scene. Therefore, LR images are aliased and shifted
with subpixel precision so multiple LR images provide the non-redundant information
but if LR images are shifted by integer unit, each image contains the same information
and thus there is no additional information that can be used to construct an HR image so
super resolution is not possible.

If there exists subpixel motions between these low resolution frames thus LR images
have different subpixel shifts (looks) from each other. In this case, the new information
contained in each LR image then SR is possible.
Figure 5: Basic premise of super resolution




*

1.3.1 Common Image Acquisition System
When we record a digital image with camera, there is a natural loss of spatial
resolution due to the optical distortions (out of focus, diffraction limit, etc.) motion blur
caused by limited shutter speed, noise occurs, and insufficient sensor density. The
recorded image, therefore suffers from these distortions.



When it comes to set of images the image acquisition system can be illustrated as
follows:


Figure 6-7: Common Image Acquisition System
Super resolution is used to reconstruct HR images from undersampled LR images,
however, it also can be used to restore HR image from a noisy,blurred images and aliased
LR images but they are not the same , SR can restore image and increase size of the
image as well.


+

1.3.2 Observation Model
We first need to establish models that describe the acquisition of the LR images.

Figure 8: Observation Model relating LR images to HR images

We can represent the observation model as

y
k
=DB
k
M
k
+n
k
for k=1,,p
when is kth observed LR image yk
D is a subsampling matrix
B
k
is a blur matrix
M
k
is a warp matrix
n
k
represents a lexicographically ordered noise vector
Most of the SR image reconstruction methods proposed in the literature consist of
the three stages illustrated in the following figure.

Registration or motion estimation, interpolation, and restoration. These steps can
be implemented separately or simultaneously according to the reconstruction methods.

Figure 9: Common Super Resolution stages

#,

1.4 Optical-flow based super resolution
Considering computational time and for more various types of transformation
super resolution with optical flow based can be a good solution here. Super resolution
optical flow based takes motion estimation information for warping an image in this step.
Our scheme will be, first, interpolate every scene in the input sequence using
chosen interpolation method[1,2,3] (interpolation method that gives maximum PSNR
results) to twice the input scenes resolution. Second, using optical flow algorithm (in this
report we use Ce Liu [23]) warping the previous and the next scenes to the current scene
and combine them with fusion. As a result, we obtain an initial approximation to the SR
image with empty gaps. Then, third, to fill the gaps, we use an inpainting technique (in
this report we regularized with Total Variation) to complete the resulting SR image.
Fourth, after we get the SR of the current scene we will warp from the current scene
using this SR image to previous scene and next scene in order to have improved neighbor
scenes. Finally we redo the second and third steps until the result is converged.




















##

The following figure illustrates our scheme.




previous scene low-resolution image current scene low-resolution image next scene low-resolution image

(1) interpolate interpolate interpolate







(2) warp to current scene warp to current scene




warped from previous warped from next


(3) fusion

(4) total variation

(5) warp to previous scene warp to next scene




Improved previous scene Improved next scene








-./0 .1234 "50/2346 784947 :2
;48297:/<8:46 "5.=4 .96 ":4/.:4
7:40 %>& <9:"? :@4 /47<?: "7
82934/=46A
;48297:/<8:46
"5.=4


#$

1.5 Keywords

Spatial resolution: a digital image is made up of small picture elements called pixels.
Spatial resolution refers to the pixel density in an image and measures in pixel per unit
area.
Aliasing: Artifacts arising from sampling and consequent loss of information cause by
sampling with too high sampling rate.
Prior information or Prior probability: Statistics the probability assigned to a
parameter or to an event in advance of any empirical evidence, often subjectively or on
the assumption of the principle of indifference.
Penalty method and Penalty function: A penalty method solving a constrained
optimization problem by a series of unconstrained problems whose solutions ideally
converge to the solution of the original constrained problem. The unconstrained problem
are formed by adding a term, called a penalty function, to the objective function.
















#%

Chapter 2
Motion Estimation

Motion estimation is an important component of Image Super resolution also in
the field of video compression as shown in figure 10,without information from motion
estimation we cannot proceed to the reconstruction step.
Motion estimation is the process to identify how the information has moved
within a sequence of images. In the other word Super resolution from multiple images or
frames is possible only in the presence of motion. Frame-to-frame motions provide
different view of the scene and motion estimation is used to detect the sub-pixel shifts
displacement between such frame and its adjacent ones. These sub-pixel shifts are
produced by the camera or by the motion of objects in the scene.
Motion estimation or Image registration for imaging super resolution is widely
considered the most influential factor when considering sources of poor performance in
super resolution. Motion estimation has to be accurate for achieving good super
resolution result. Different techniques have been developed to solve the estimation
problem. Motion estimation mainly compute shifts (displacements of pixels) or
rotation(angle rotated of pixels) or both between two frames.


Figure 10: Overview of common super resolution algorithms

Motion estimation plays a big role in super resolution algorithm. In the end of this
chapter, we will show the effect of mis-registeration on result of HR image.









#&


2.1 Preliminaries

2.1.1 Motion type

! B/.97?.:"29 C !
!
!
!
!
D E F
!
!
!
!

!
!
!
D
! G"5"?./":H C !
!
!
!
!
D! 7 F
!"# !!!
!!"# !!!

!"# !!!
!"# !
D F
!
!
!
!
!
!
!
D
! IJJ"94 C !
!
!
!
!
D E F
!"
!
!"
!
K
!"
!
!"
!
K
!
!
D
! L252=/.0@H C !
!
!
!
!
D E
!
!
F
!"
!
!"
!
K
!"
!
!"
!
K
!
!
D> M E !"
!
K !!
!
K "

2.1.2 Intensity Conservation Assumption

The total intensity is conserved from frame to frame. This assumption holds under
ideal conditions, i.e. no large motions, consistent lighting conditions and no occlusions.
We can express intensity conservation as follows:

I
1
(x,y) = I
2
(x+u,y+v)+r+g
r ~ N(0,!
2
), g ~ U(-1,1)
Noise r,outlier g (occlusion,lighting change)


2.2 Motion Estimation Methods

In this part we will focus on two motion estimation techniques for being
commonly used. These are block-matching method and Optical flow method. Block-
matching techniques are slower but more accurate. Optical flow methods, on the other
hand, are less accurate but faster and more suitable for obtaining dense motion vectors
where the motion is small.

Lets denote by a pixel in the LR image.
x and y represent the spatial coordinates in the image plane coordinate system.
MxN is the size of the LR image.
m x n is the size of the block.
! #
!
!
!
!
!
is the total number of blocks in the image.

$% #
!
!
and $& #
!
!
are the number of blocks in the horizontal and vertical axis.
!
!
(!
!!
!
!
) is a block in the LR image.
!
!
is the notation for an entire image.
!
!
denotes the image we want to improve its resolution.


#'

!
!
('()) denotes HR image pixel.
P is the magnification factor.
PMxPN is the size of on HR image.
!
!
($'($)) is a HR image block
PmxPn is the size of each block
!
!
is the HR image we seek to estimate
!
!!!
!!!"
.96 !
!!!
!!!"
are the horizontal and vertical components of displacement of a
pixel(x,y) from the k-th LR image into the l-th image.
(!
!!!
!!!"
, !
!!!
!!!"
) are the motion vectors for HR images
then,

To obtain a LR image !
!
! the corresponding HR image!
!
! we will decimate the HR
image by taking one pixel each P in rows and columns. This process is depicted in
Figure11. Mathematically we have

and our goal is to estimate f
k
from(g
k-2
,,g
k+2
)
2.2.1 Block-Based Motion Estimation Algorithms
Block-matching Motion Estimation by least square estimate.

Block-matching algorithms is one of the most popular approach for motion
estimation between images due to its low computational cost involved. They simply
divide one frame into blocks of size m x n. These algorithms search for each block in
certain area of another frame in the sequence. They are then assuming that every pixel in
a block has the same movement.
- Choosing a large block size implies less motion vectors to calculate, but the
probabilities for each block to contain objects with different movements are higher.
- Choosing a small block size will increases the computation, but it is also more sensitive
to the noise present in the image.

Matching Criteria
Mean of Squared Error (MSE)

The intensity value of the pixel at coordinate(n1,n2) in the frame k-th is given by
s(n1,n2,k) of block size N x N. (i,j) is a displacement.


#(

MSE is computed for each displacement position (i,j)within a specified search
range in the reference image and the displacement that gives the minimum value of MSE
is the motion vector and given by


Figure 11: Procedure to obtain a LR image sequence from a HR image sequence
by down sampling

Sum of Absolute Differences (SAD)

We may use SAD in order to reduce the computational cost because it use only
addition and subtraction


we are considering low resolution data g for the motion estimation. Low values of this
measure show high correlation between the current block and the displaced block
according to a particular motion vector(!
!!!
!!!"
, !
!!!
!!!"
).The block that turns out to be the
most similar to the current block(bx,by)in the reference frame, is given by the candidate
vector with the minimum SAD:

for all motion vectors in the search area A.


#)

Searching for the best block in every position of the search window is called full
search block-matching. Faster algorithms exist that decrease the computational load, but
they usually degrade the quality of the estimations. For this report we use Full search
motion estimation.
Here are the commonly used steps:
Step1:Calculate the best motion vector at pixel precision.
Step2:Search for the best half-pixel precision motion vector at eight half-pixel positions
around the point of minimum given in step1. These new positions contain interpolated
values.

Figure 12: Block-matching frame division in blocks of size m x n. The reference block
that is the most similar to the current block is found within the search defined by the
parameter p.
Step3: Compare the current block with eight new interpolated blocks around the best
half-pixel position. These blocks contain values that are interpolated from pixel precision
image data and half-pixel precision interpolated data.


2.2.2 Optical Flow-Based Motion Estimation Algorithm

Optical flow is used to compute the motion of the pixels of an image sequence. It
provides a dense pixel correspondence i.e. determine where the pixel of an image at time
t are in the image at time t+1.This method is good when motion is small.
Since the Lucas-Kanade algorithm was proposed in 1981 for image alignment it
has been widely used in computer vision for multiple tasks. One of them has been to
compute the motion field between a pair of images within an image sequence or video.
This technique calculates motion vector estimates(!
!!!
!!!"
, !
!!!
!!!"
) between LR images l-th
and k-th,which satisfy the optical flow equation with the minimum pixel to pixel variation
in the velocity field.There are multiple variations in the literature of this method,
depending on the purpose, we will use the following algorithm.
The motion vector for each pixel(x,y) is defined by !
!!!
!"
!!! !!= (!
!!!
!!!"
, !
!!!
!!!"
) We
will center a window of size m x n on each pixel, the error to be minimized in the optical
flow equation will be as follows


#*


Using a second-order Taylor expansion for the second term of Equation above we have
(**)

and

To obtain more accurate motion vectors, two processes has been added:
Motion vectors refinement
The refinement is based on an iterative estimation of the motion vectors. Once the
initial motion vector are calculated the image is compensated via this motion field and
equation (**) is again applied for each pixel.

Hierarchical approximation
The hierarchical improvement is achieved using a pyramid of three levels to
calculate the motion vectors in a coarse to fine approximation.

Coarse-to-fine refinement
-Lucas-Kanade is a greedy algorithm that converges to local minimum.
-Initialization is crucial:if initialized with zero,then the underlying motion must be small.
-If underlying transform is significant, then coarse-to-fine is a must.


#+


Figure 13:Coarse-to-fine optical flow estimation

The algorithm, including these improvements, can be summarized in the
following steps:
Step1:Downsample the images by a factor of 2 and 4 to build the hierarchical pyramid.
Step2:Compute the motion field for the lowest level in the pyramid by minimizing the
optical flow equation for each pixel as shown in equation(**).
Step3:Iterate over the motion vectors to refine.
Step4:For each higher level in the pyramid
Step4.1:Propagate the motion vectors to the higher level.
Step4.2:Iterate to refine.

2.2.3 Frequency Domain Motion Estimation Algorithms[3],[4]

The frequency domain approach makes explicit use of the aliasing that exists in
each LR image to reconstruct an HR image. Tsai and Huang first derived a system
equation that describes the relationship between LR images and a desired HR image by
using the relative motion between LR images. The frequency domain approach is based
on the following three principles:
i) the shifting property of the Fourier transform
ii) the aliasing relationship between the continuous Fourier transform(CFT) of an
original HR image and the discrete Fourier transform(DFT) of observed LR images.
iii) and the assumption that an original HR image is bandlimited.






$,

For example, let us assume that there are two 1-D LR signals sampled below the Nyquist
sampling rate. From the above three principles, the aliased LR signals can be
decomposed into the unaliased HR signal as shown inFigure 14.


Figure 14: Aliasing relationship between LR image and HR image.

The basic premise of these techniques is the idea that if a sequence of images,
thought of as a 3-D function [in two-dimensional(2-D) space and time]. Contains a
uniformly translating pattern or object, then the energy of the 3-D fast Fourier
transform(FFT) of this function will be mostly concentrated along a plane through the
origin whose orientation is related to the velocity of the moving pattern. Hence, by
computing the 3-D FFT and then finding one or more planes with strong energy
concentration, one can estimate the desired velocity.
An important advantage of this technique is that because of the linear
superposition of the Fourier transform, the existence of multiple superimposed motion
will be manifested in the FFT magnitude simply as energy concentrations along more
than one plane through the origin.














$#

2.3 Effects of Motion Estimation on Performance of Super Resolution.

As mentioned in the beginning of this chapter that motion estimation quality has
impact on result of super resolution. We will show the result of super resolution with
registration error.


Figure 14: Super resolution with mis-registration

We can conclude that the motion estimation technique must be as accurate as
possible. For the past literatures a little attention has been paid to the performance of
motion estimation techniques when used in SR problems. Schultz et al. [6] compared
several sub-pixel motion estimation techniques into the Bayesian multiframe
enhancement algorithm, eight parameter projection motion model,block-matching and
Horn-Schunck optical flow estimation. They concluded that each technique has its own
advantages and disadvantages depending on the kind of motion and the image
characteristics.




$$

Chapter 3
Reconstruction
3.1 Optical-flow based super resolution reconstruction

3.1.1 Image Warping
Since we know the information between scenes (registration of input image
sequence) by motion estimation .We can use it to combine input sequence images after
interpolation by warping them to match the input reference image. Image warping is the
process of digitally manipulating an image such that any shapes portrayed, which have
been significantly distorted in the image. Warping may be used for correcting image
distortion for motion compensation.
Performing this warp involves interpolating the input images, a task that is
normally performed using one of the standard algorithms, such as nearest-
neighbor,bilinear, or cubic-spline interpolation.


Figure 15: (left) input low resolution images, and (right) combination of LR images to
create the HR image.





$%

3.1.2 Image Fusion.

The warped images( images after interpolation and warping) can be regarded as
multiple estimates of an underlying high resolution image. From these warped images,
we need to combine them. The simplest approach is to average the estimates, taking
either the mean, the median , or a robust mean in this report we use mean to average and
fuse the warped images.

3.1.3 Image Inpainting using Total Variation Regularization.

Image inpainting refers to the process of filling-in missing data or damged part in
a designated region of the visual input.
Regularization is used to measurement information with some general prior
information about the desirable SR solution. It takes an important role in inpainting like
in super resolution as well.
The Total Variation(TV) norm is a gradient penalty function. The TV criterion
penaliz

3.1.4 Image restoration

Figure 16:Deblurring block. Image restoration consists of deblurring and
denoising.
This process of super resolution algorithms can be done by apply deblurring and
denoising.

Deblurring

With Camera motion causes the blur to occurred and considered to be a serious
problem in Super Resolution. This problem occurs when low ambient light conditions
prevent an imaging system from using sufficiently short exposure times.

Power Spread Function(PSF) is a filter that models the image blur effect.
There are two types of blur in all digital imaging systems
1)the optical blur : specifies the fine-resolving capability of the optics.
2)the sensor integration blur : caused by the accumulation of incoming light over a finite-
size sensor element.


$&



Deblurring allows the recover of an image with a good resolution. Hence, the
estimate of the PSF is essential to the application of these algorithms.

Several super resolution algorithms have included an optional post-processing
step to deblur the super resolution image. In this report we use Wiener Filter for
deblurring.

Denoising
Denoising is the process that removes the image noise,which is a random
variation of brightness in image.Denoising process plays a role in super resolution as well
as deblurring.In this report median filter is used for denoising.

















$'

Chapter 4
Result
Qualitative result
Image Sequence I: Plant



low resolution images size 128*128pixels

HR images size 256*256 pixels


original image after interpolate by bilinear method

our method our method after wiener filter


$(


Image Sequence II: Baby

low resolution images size 128*128pixels
HR images size 256*256 pixels


original image


our method our method after wiener filter










$)





Image Sequence III: Bowling


low resolution images size 128*128pixels

HR images size 256*256 pixels

original image


our method our method after wiener filter





$*


Image Sequence IV: Flower Pots


low resolution images size 128*128pixels

HR images size 256*256 pixels


original image


our method our method after wiener filter








$+

Quantitative result

Image
set
Scene PSNR in dB with method
Bicubic Nearest Bilinear NEDI Our
method
Our method after
wiener filter
Plant 1 27.9231 26.5548 28.1876 29.1386 29.3183 29.2522
2 27.8827 26.5044 28.1494 29.1281 29.4855 29.3650
3 27.8674 28.4846 28.1301 29.0807 29.3487 29.3026
4 27.8775 26.5078 28.1386 29.0734 29.3883 29.2945
5 27.8967 26.5099 28.1604 29.1597 29.3767 29.3270
Baby 1 33.8193 32.4879 34.0689 34.9001 35.0704 35.2489
2 33.9751 32.6006 34.2318 35.0397 35.2206 35.3284
3 33.9529 32.5996 34.1998 35.0523 35.3768 35.4869
4 34.1184 32.7626 34.3760 35.1955 35.3543 35.5016
5 34.2442 32.8791 35.4867 35.3660 35.3851 35.5319
Bowling 1 33.1791 31.5532 33.3628 34.9857 32.1387 32.7856
2 33.1458 31.5250 33.3173 34.9343 31.9131 32.5933
3 33.0885 31.4674 33.2478 34.9311 31.7601 32.4243
4 33.1957 31.5903 33.3347 35.1195 31.9877 32.6261
5 33.1906 31.5863 33.3255 35.0714 32.0564 32.7074
Pots 1 33.6402 31.8266 33.7436 35.5478 32.6208 33.8836
2 33.9926 32.1868 34.0338 35.7892 32.6648 33.9363
3 33.6477 31.8356 33.7206 35.6193 32.5010 33.8138
4 33.5858 31.7728 33.6784 35.4704 32.9696 34.2235
5 33.5723 31.7598 33.6545 34.8822 32.8238 34.1682
Lamps 1 34.0099 32.3928 34.1879 35.9281 32.8042 34.2530
2 34.0681 32.4370 34.2582 36.0965 32.8904 34.3201
3 34.1466 32.5403 34.2767 36.0286 32.8309 34.3136
4 33.9895 32.3590 34.1925 36.2382 32.8455 34.2982
5 34.2015 32.5790 34.3640 36.4198 32.7324 34.3120


Table 1: PSNR results of super resolution methods.








%,

Chapter 5
Conclusion& Future works.
Super Resolution algorithms are made to enhance image quality by increasing
resolution and also recovery image from blur and noise. The algorithms performances
are limited under availability of input low-resolution images (i.e. better performance with
more input LR images) but still can be considered as okay because we reduced
production cost and still be able to get better image.
Optical-Flow based Super resolution based on works of Lucas-Kanade and Horn-
schunck. The algorithms have flexibility to non-planar motion(i.e. Human faces) and use
less computational time than other approach. Our Optical-flow based super resolution
have a well performance, as you can see in chapter 4 the experiment result shown by
PSNR result is higher than other interpolation methods for some set of image and slightly
has lower PSNR in some set of image as well.
Our super resolution method was applied to gray scale image only in the future
we will try to improve the algorithms to be able to handle color image as well.













%#


Bibliography:

[1] S.Baker and T.Kanade, Super Resolution-Optical Flow, 1999

[2] Aggelos K.Katsaggelos, Rafael Molina, Javier Mateos, Super Resolution of Image
and Video, Department of Electrical Engineering and Computer Science,Northwestern
University, Evanston, Illinois, USA, 2007

[3] P. Milanfar, Projection-based, frequency-domain estimation of super-imposed
translational motions, J.Opt. Soc. Amer. A, vol.133, pp.2151-2161, Nov. 1996

[4] P. Vandewalle, S. Susstrunk, M.Vetterli, A Frequency Domain Approach to
Registration of Aliased Images with Application to Super-Resolution Ecole
Polytechnique Federale de Lausanne(EPFL),Lausanne,Switzerland.

[5] D. Rivero Alfonso, Image Reconstruction and Restoration for Optical Flow-based
Super Resolution,Master Thesis, Universitat Politecnica de Catalunya-ETSETB
Barcelona, 2012

[6] D. Barreto, L.D.Alvarez, J.Abad, Motion Estimation Techniques in Super-
Resolution Image Reconstruction. A Performace Evaluation, Department of Computer
Science and Artificial Intelligence, University of Granada,Spain,2005
[7] Ce Liu, Motion Estimation I&II, Microsoft Research New England, 2005
[8] Xin Li, Michael T. Orchard, New Edge-Directed Interpolation, IEEE Transactions
on Image Processing, Vol.10,No.10,October 2001
[9]Nhat Xuan Nguyen, Numerical Algorithms For Image Superresolution, Scientific
Computing And Computational Mathematics, Stanford University, July 2000
[10] S.C.Park, M.L.Park, M.G.Kang, Super-resolution image reconstruction : a technical
overview, IEEE Signal Procesing Magazine, vol.20, no.3,pp.21-36,2003.
[11]Wan-Chi Siu and Kwok-Wai Hung, Review of image interpolation and Super-
resolutionThe Hong Kong Polytechnic University, Hong Kong, China.
[12]H. Hirschmller and D. Scharstein. Evaluation of cost functions for stereo matching.
In IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR 2007), Minneapolis, MN, June 2007.


#

Appendix
MATLAB code
%Assigning global values
[nRow, nCol] = size(curr_orig);

blksize = 32;
trans_mode = 'BWHT';
subrate =0.25;
N= nRow*nCol;
M = round(subrate*N);
rand_vect = randperm(N); % Reordering vector
rand_vect = sort(rand_vect)';
select_vect= rand_vect(1:2:end); %current one

%modified here
MAT=vec2mat(select_vect,nRow/2);
select_vect=MAT(1:2:end);
select_vect=select_vect(:);
select_vect=sort(select_vect);




A = @(z) blk_f1d_MT(z, select_vect, rand_vect, trans_mode, blksize);
A_T = @(z) blk_t1d_MT(z, N, select_vect, rand_vect,trans_mode,
blksize);

%downsampling HR images sequence into LR images sequence
[prev_LR ,curr_LR, next_LR] = downsamplingHRs(prev_orig, curr_orig,
next_orig);


%interpolate them with several methods
%1st-Classical method (bicubic,bilinear,nearest)
[prev_bicubic_HR,prev_bilinear_HR,prev_nearest_HR] =
classicalinterp(prev_LR);
[curr_bicubic_HR,curr_bilinear_HR,curr_nearest_HR] =
classicalinterp(curr_LR);
[next_bicubic_HR,next_bilinear_HR,next_nearest_HR] =
classicalinterp(next_LR);
%2nd-NEDI
prev_NEDI_HR=nedi(prev_LR);
curr_NEDI_HR=nedi(curr_LR);
next_NEDI_HR=nedi(next_LR);

%Evaluate the results of interpolation by PSNR

%Previous scene
prev_PSNR_bicubic=PSNR1(prev_bicubic_HR,prev_orig);
prev_PSNR_bilinear=PSNR1(prev_bilinear_HR,prev_orig);
prev_PSNR_nearest=PSNR1(prev_nearest_HR,prev_orig);


$

prev_PSNR_nedi=PSNR1(prev_NEDI_HR,prev_orig);

%find maximum PSNR
prev_PSNR=[prev_PSNR_bicubic,prev_PSNR_bilinear,prev_PSNR_nearest];
[max_prevPSNR i]=max(prev_PSNR);
if i == 1
display(max_prevPSNR ,'previous scene maximum by bicubic
interpolation in dB');
elseif i == 2
display(max_prevPSNR,'previous scene maximum by bilinear
interpolation in dB');
elseif i == 3
display(max_prevPSNR,'previous scene maximum by nearest
interpolation in dB');
% elseif i == 4
% display(max_prevPSNR,'previous scene maximum by New Edge-
Directed interpolation in dB');
end
%choose output from the method that gives maximum PSNR
if i == 1
interpHR_prev=prev_bicubic_HR;
elseif i == 2
interpHR_prev=prev_bilinear_HR;
elseif i == 3
interpHR_prev=prev_nearest_HR;
% elseif i == 4
% interpHR_prev=prev_NEDI_HR;
end

%Current scene
curr_PSNR_bicubic=PSNR1(curr_bicubic_HR,curr_orig);
curr_PSNR_bilinear=PSNR1(curr_bilinear_HR,curr_orig);
curr_PSNR_nearest=PSNR1(curr_nearest_HR,curr_orig);
curr_PSNR_nedi=PSNR1(curr_NEDI_HR,curr_orig);

%find maximum PSNR
curr_PSNR=[curr_PSNR_bicubic,curr_PSNR_bilinear,curr_PSNR_nearest];
[max_currPSNR j]=max(curr_PSNR);
if j == 1
display(max_currPSNR ,'current scene maximum by bicubic
interpolation in dB');
elseif j == 2
display(max_currPSNR,'current scene maximum by bilinear
interpolation in dB');
elseif j == 3
display(max_currPSNR,'current scene maximum by nearest
interpolation in dB');
% elseif j == 4
% display(max_currPSNR,'current scene maximum by New Edge-
Directed interpolation in dB');
end
%choose output from the method that gives maximum PSNR
if j == 1
interpHR_curr=curr_bicubic_HR;
elseif j == 2
interpHR_curr=curr_bilinear_HR;
elseif j == 3
interpHR_curr=curr_nearest_HR;


%

% elseif j == 4
% interpHR_curr=curr_NEDI_HR;
end


%next scene
next_PSNR_bicubic=PSNR1(next_bicubic_HR,next_orig);
next_PSNR_bilinear=PSNR1(next_bilinear_HR,next_orig);
next_PSNR_nearest=PSNR1(next_nearest_HR,next_orig);
next_PSNR_nedi=PSNR1(next_NEDI_HR,next_orig);

%find maximum PSNR
next_PSNR=[next_PSNR_bicubic,next_PSNR_bilinear,next_PSNR_nearest];
[max_nextPSNR k]=max(next_PSNR);
if k == 1
display(max_nextPSNR ,'next scene maximum by bicubic
interpolation in dB');
elseif k == 2
display(max_nextPSNR,'next scene maximum by bilinear
interpolation in dB');
elseif k == 3
display(max_nextPSNR,'next scene maximum by nearest
interpolation in dB');
% elseif k == 4
% display(max_nextPSNR,'next scene maximum by New Edge-Directed
interpolation in dB');
end
%choose output from the method that gives maximum PSNR
if k == 1
interpHR_next=next_bicubic_HR;
elseif k == 2
interpHR_next=next_bilinear_HR;
elseif k == 3
interpHR_next=next_nearest_HR;
% elseif k == 4
% interpHR_next=next_NEDI_HR;
end



%Apply Optical Flow
%assigning values of para
alpha = 0.092;
ratio = 0.75;
minWidth = 20;
nOuterFPIterations = 7;
nInnerFPIterations = 1;
nSORIterations = 30;
[nRow, nCol] = size(curr_orig);
para =
[alpha,ratio,minWidth,nOuterFPIterations,nInnerFPIterations,nSORIterati
ons];

%using Coarse2finetwoframe warp 2 images to make 2 new current images
base
%on motion between the previous scene and the current scene, motion
between


&

%the next scene and the current scene.

for k = 1 : 1
[vxl,vyl,warpedbyprev] =
Coarse2FineTwoFrames(interpHR_curr,interpHR_prev,para);
[vxl,vyl,warpedbynext] =
Coarse2FineTwoFrames(interpHR_curr,interpHR_next,para);
warped_image = (warpedbyprev + warpedbynext)/2;

y = A(curr_orig(:));
Xpred = warped_image(:);
y_pred = A(Xpred);

r = y - y_pred;
r_rec = tval3_residual(r,A,A_T, nRow, nCol);
img_rec = r_rec + warped_image;

reconstructedImg = curr_orig;
reconstructedImg(2:2:end,:) = img_rec(2:2:end,:);
beforeimprove= PSNR1(curr_orig, reconstructedImg)
% interpHR_curr=reconstructedImg;
%choose wiener filter block that gives maximum PSNR
afterwiener2=wiener2(reconstructedImg,[2 2]);
PSNRAfterwiener2= PSNR1(curr_orig,afterwiener2);
afterwiener3=wiener2(reconstructedImg,[3 3]);
PSNRAfterwiener3= PSNR1(curr_orig,afterwiener3);
afterwiener=[PSNRAfterwiener2,PSNRAfterwiener3];
[max_afterwiener ,t]=max(afterwiener);
if t == 1

afterwiener=afterwiener2;
elseif t == 2
afterwiener=afterwiener3;
end
PSNRAfterwiener= PSNR1(curr_orig,afterwiener)

figure;imshow(curr_orig),figure;imshow(reconstructedImg),figure;imshow(
afterwiener);


end