Recent trends on color guided thermal 2019-2020

image super resolution

Chapter 1
Introduction
Infrared (IR) cameras (also called thermal cameras) are passive sensors, which have a
spectral sensitivity to the 7µm to 14µm wavelength band. They are sensitive to the
radiation emitted, transmitted, and reflected by an object in the scene. The special
properties of the infrared cameras give us a new perspective to solve those vision
problems which are difficult or complex for conventional visible images. They have been
widely used in many fields: transportation, automation, law enforcement, firefighting,
research, building, security, etc. For example, in industry, workers use the IR cameras to
detect the stress and fault of the machine. IR sensors are even embedded on some mobile
phones, expanding IR imaging to people’s daily life. The sensor are supposed to use the
thermal map as augmented reality that help people find unusual stuffs at home as well as
act as a night vision camera.
However, the IR camera usually has low resolution, mostly only 160×120, and 320×240,
which is limited by the finite size of the infrared focal plane, the optics of the camera and
cooling of the sensor. Though to improve imaging device to obtain a high resolution IR
image, it would dramatically increase the production cost. Little work has been done to
enhance the low resolution IR imaging, either hardware or software. This limits the work
in high-level image processing tasks which require good quality of the imagery. The
application and user experience of the IR cameras are largely constrained due to their low
resolution while the color cameras are developing rapidly. A high quality visual camera
has become the common equipment on many devices on account of its giveaway prices
and high resolution. Thus, to reconstruct the low-resolution IR images guided by the
visible-light camera. Super-resolution is a technique to allow the construction of a higher
resolution image of an area, based on the merging of information from other related
images. It is related to many other image processing problems, such as image coding,
reconstruction, error concealment, and deblocking.
The fact that a registered high-quality texture image can provide significant information
to enhance the infrared image due to their strong correlation. The key method is the use
of a guided filter in the correlated region between IR image and color image and
construct a cost volume of IR image values probability based on the input IR image. A
guided filter is then applied to the cost volume in the correlated region. After that, two
steps including a best cost selecting and sub-pixel refinement are taken to produce a
refined IR image. The output image is finally got after a outlier detection. The algorithm
works fast and turns out to be suitable for IR image super resolution.

Dept. of ECE, S.I.T., Tumakuru 1

Recent trends on color guided thermal 2019-2020
image super resolution

1.1 Organization of the report

This report is organized as follows: Chapter 2 deals with the setup of a prototype of
multi-sensor system which realize the calibration between the two different sensors.
Chapter 3 presents the methodology proposed to enhance an IR image combined with a
HR color image using a guided filter. Chapter 4 deals with the experimental results,
followed by a conclusion in Chapter 5.

Dept. of ECE, S.I.T., Tumakuru 2

Recent trends on color guided thermal 2019-2020
image super resolution

Chapter 2
In this Chapter, it introduces an apparatus and methodology to get the registered IR and
visual image sets. Firstly, describe system settings. Then, to show how to calibrate the
two cameras. Finally leverage a well performed algorithm to register images from two
different sensors, including the IR sensor and the RGB sensor. With this system,
contribute several video sequences for further research. Dataset provide HR color video
and corresponding IR map, including both indoor and outdoor scenes

2.1 System setting

Fig. 2.1(a) shows the hardware configuration. images are captured with a special
apparatus printed by a 3D printer, which combine an infrared camera (Magnity MAG62)
with an industrial visible camera (Kedacom IPC120). The infrared camera has a much
lower resolution (only 640×480) than the RGB camera (1280×960 pixels). The selected
IR camera can capture LWIR wavelength between 7.5 to 14 µm. To make two sensors
get the same image, gold dichroic beam- splitter is placed between them. The beam-
splitter allows the visible radiation to be projected to the visible camera, while reflecting
the IR radiation from the scene towards the IR camera.

(a)

Figure 2.1 (a) The structure diagram of the system

Dept. of ECE, S.I.T., Tumakuru 3

Recent trends on color guided thermal 2019-2020
image super resolution

2.2 Camera Calibration and Color-IR Registration

The multi-sensor system setup makes the two cameras share the same scene. However,
the lens used by both sensor are different, which means their parameters are dissimilar. so
calibrate the two cameras before aligning image pairs. We can calibrate our two cameras
by taking advantage of mature algorithm proposed by Zhang. It takes easy to finish the
visual camera calibration, but remains difficulty in calibrating the IR camera due to its
unidentification to an ordinary chess. In order to solve the problem, we make a special
”chess” printed by a 3D printer as shown in Fig. 2.2(a). We fill the chess with hot water
every other lattice. So you can see the IR image like Fig. 2.2(b) with high contrast. Now
Zhang’s algorithm performs well. The image pairs from two different sensors are not
aligned for pixels , which has a direct influence on the performance of the super
resolution algorithm. The method proposed by Han with good performance is used to
solve this problem. Fig. 2.3 shows several images and their corresponding IR images in
the datasets

(a) (b)

(c)
Figure 2.2 (a) Chess printed by 3D printer (b) the thermal image of chess after filled
with water (c) Some examples from datasets

Dept. of ECE, S.I.T., Tumakuru 4

Recent trends on color guided thermal 2019-2020
image super resolution

Chapter 3 The proposed approach

Figure 3 shows the overview of the framework of the proposed approach. First, up-
sample the LR infrared image IL(e.g 160×120) to the same size with the HR color image
H(e.g. 640×480) using bicubic interpolation, and keep it as I0. And calculate the
correlated region CR between the suitable channel Hr of the image and IR image. Next is
an iterative module. A cost volume Ci is built based on the current infrared Ii, then a
guided filter is performed throughout each slice of the cost volume to produce the new
cost volume Ccw i in the correlated region. The refined infrared image Ii+1 is calculated
by taking the winner-takes-all approach on Ccw i and a sub- pixel estimation afterwards.

Figure 3 The flow of proposed algorithm

Dept. of ECE, S.I.T., Tumakuru 5

Recent trends on color guided thermal 2019-2020
image super resolution

3.1 Cost volume

A coarse cost volume is first built to preserve the sub-pixel accuracy of the input IR
image based on current IR image estimation Ii. As the differences get large, the cost
function should be- come constant in order to make candidate IR values vary a lot
because the current values are not necessarily correct. The truncated quadric model is one
of available function. Thus the cost function can be defined as following:

𝐶𝑖 (𝑥, 𝑦, 𝑑) = min⁡(𝜂 ∗ 𝐿, (𝑑 − 𝐼𝑖 (𝑥, 𝑦))2 ) (2, 1)

3.2 Guidance selection

The basic idea of this report is based on the fact that the infrared image discontinuities
often co-occur with color or brightness changes within the associated camera image of
the same scene. A color camera is a combination of three sensors: red, green, and blue.
Different channels have different correlation with IRimage. To verify this ,we calculate
the PSNR and SSIM between different channels and the corresponding IR images in the
database. Fig. 3.1(a) shows the channels histogram that IRimage are most relevant to over
200 image pairs. We can see that about 63% of IR images have the strongest correlation
with the associated red channel with green 12% and blue 25% respectively. Thus in this
report, we choose the most correlated channel instead of the all channels for a better
guidance.

(a)
Figure 3.1(a) Histogram of channels that most relevant to IR sensor

Dept. of ECE, S.I.T., Tumakuru 6

Recent trends on color guided thermal 2019-2020
image super resolution

3.3 The Correlated Region

Though most discontinuities in the infrared image correspond to discontinuities in the
color image of the same scene, they have much difference in some situations. For
example, the visible camera can see the texture of a book’s cover, which the thermal
imager is blind to. To avoid wrong texture transfer, we only apply guided filter in their
correlated region. The correlated region could be calculated by their cross-correlation in a
small patch. If the value is bigger than the threshold T, the point is regard as one point of
correlated region. The others is in uncorrelated region.

3.4 Guided Filter in Correlated Region

The guided image filter is a novel explicit image filter. It not only has good edge-
preserving smoothing properties like the bilateral filter, but also makes the filtering output
more structured and keeps more details than other edge-preserving filters. Compared with
the traditional edge-preserving method- s, the guided image filter works much faster.
Thus we adopt the guided filter instead of bilateral filter used in report. Here we put the
suitable channel of color image Hr as a guidance, and filter the cost volume Ci only in
region CR. The output can be computed by

𝑐𝑤 1
⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡𝐶𝑖(𝑖) =|𝜔| ∑𝑘:𝑖∈𝜔𝑘(𝑎𝑘 𝐻𝑟(𝑖) + 𝑏𝑘 ) (2, 2)

∑𝑖𝜖𝜔𝑘 𝐻𝑟(𝑖) 𝐶𝑖(𝑖) −𝜇𝑘 𝑝𝑘

𝑎𝑘 = (2, 3)
𝜎𝑘2 +∈

𝑏𝑘 =𝑝𝑘-𝑎𝑘 𝜇𝑘 (2, 4)

Here,⁡𝜇𝑘 and 𝜎𝑘2 are the mean and variance of I in 𝜔𝑘 , |w| is the number of pixels in
1
window 𝜔𝑘 , and 𝑝𝑘 =|𝜔| ∑𝑘∈𝜔𝑖 𝑎𝑘 the mean of p in 𝜔𝑘 . After this, we can get sharp and
right edges while keeping the true values in smooth areas.

Dept. of ECE, S.I.T., Tumakuru 7

Recent trends on color guided thermal 2019-2020
image super resolution

3.5 Sub-pixel Estimation

As we mentioned in Part A, the discontinuity of cost function and limited search range
result in the discontinuity of IR images. In order to eliminate this effect, quadratic
polynomial interpolation is adopted here to approximate the cost function between three
discrete IR image candidates: d, 𝑑− and 𝑑+ . d is the discrete depth with the minimal cost,
𝑑− = d−1, and 𝑑+ . = d + 1.

f(x)=a𝑥 2 +bx+c (2, 5)

f(𝑥𝑚𝑖𝑛 ) is the minimum of function f(x). The parameters a, b and c of the function f(x)
can be calculated given d, f(d),f(𝑑− ), f(𝑑+ ). So xmin can be calculated:

𝑓 (𝑑+ )−𝑓(𝑑− )
𝑥𝑚𝑖𝑛 =d-2(𝑓(𝑑 (2, 6)
+ )+𝑓(𝑑− )−2𝑓(𝑑− ))

3.6 Outlier detection

Outlier detection is used here to solve the problem that sometimes black points may
appear where tiny edges only exist in IR image. With the iteration number increasing, the
outlier points will disappear. But running time increases a lot and some edges become
smooth, which we do not like to see. In fact we only have to process those uncomfortable
points in the output image instead of the whole one. So here we use outlier detection
method to find the incorrect points and then we adopt a median filter around them to
eliminate outlier points.

Dept. of ECE, S.I.T., Tumakuru 8

Recent trends on color guided thermal 2019-2020
image super resolution

Chapter 4 Results
We test our approach using the real world examples collected in our dataset including
indoors and outdoors images. Like the other super-resolution methods, we use the
original IR image in our dataset as the ground truth. And we prepare our test in the
following manner: An input sequence is blurred using a 3×3 uniform mask, decimated by
a factor of 1 : 4 (in each axis), and then contaminated by an additive noise with std = 2.
For the main parameters in our algorithm, we set them as: ε = 0.012, η = 0.5, L = 20000,
T = 0.35 and k = 6 respectively, which makes the best performance in our experiments.
The system configuration for experiments is Quad-Core 3.4GHz CPU, 4GB RAM. We
implement our algorithm via Matlab. The bicubic interpolator, joint bilateral upsampling,
are used for comparison. Three experiment results in three different scenes are listed in
the Fig. 4.

Figure 4: From left to right are the suitable channel images, the ground truth IR
images, the bicubic interpolation results, the result using JBU and the results using
proposed algorithm. The result images has been up sampled by a factor of 1:4
(in each axis).

Dept. of ECE, S.I.T., Tumakuru 9

Recent trends on color guided thermal 2019-2020
image super resolution

We can see that algorithm performs well in all the situations. Bicubic works well in
smooth areas but smooth the discontinuous areas. The guided filter sharpen the IR image
edges in true place by using the information provided by the aligned color image. It is
only applied in the correlated region to avoid wrong edges transfering. After processing,
the edges of the enhanced IR images are sharpen, and the added noise is removed. More
importantly, the over texture and over smooth problem is avoided.

The performance of algorithm is listed in Table.1.

Dept. of ECE, S.I.T., Tumakuru 10

Recent trends on color guided thermal 2019-2020
image super resolution

Chapter 5 Conclusion

IR imaging is playing an important role in industry. However, the IR cameras usually

support low resolution for technical reasons. In this report, use of visible camera as a
guidance to super resolution the IR images.a prototype of the IR-Color multi-sensor
imaging system and construct a dataset including videos collected from different scenes
for further research. Then algorithm is presented which is suitable for this kind of super
resolution problem with guidance images. Then algorithm tested on the collected dataset.
The experiment results indicate that algorithm performs well, and avoid the over-texture
problems that traditional solutions suffer from.

Dept. of ECE, S.I.T., Tumakuru 11

Recent trends on color guided thermal 2019-2020
image super resolution

Bibliography

Dept. of ECE, S.I.T., Tumakuru 12