Image fusion-based video deraining using Fusion: In the proposed method, fusion is realised on both the diction-

sparse representation aries and the sparse coefficients according to different fusion rules.

Zetian Mi, Jinxia Shang, Huan Zhou and Minghui Wang✉

By disregarding the ‘rain component’ but keeping the ‘non-rain com-

ponent’ only, results reconstructed by the conventional morphological
component analysis decomposition-based rain removal algorithms lose
a lot of detail information. On the basis of the limitation, an image
fusion strategy is introduced to remove rain from a video. The final
rain-free frame is recovered by employing the fused coefficients and a b
the fused dictionary. Experimental results demonstrate that the pro-
posed method can efficiently remove rain streaks, while at the same
time preserve more detail information.

Introduction: Rain videos consist of a large number of rain drops,

moving at high velocities and time-varying dynamically. Each drop
refracts and reflects light from the environment, which causes sharp c d
intensity changes in the image sequences. In most literatures, temporal
and chromatic properties of rain are exploited to detect and remove Fig. 1 Image fusion-based rain removal on image sequence
rain streaks from videos. Since rain produces temporal variations, a a Current frame Ik
b Optimal warped frame Ĩ k
certain number of consecutive frames are required to detect rain c Difference between (a) and (b)
streaks, which increases the cost of the system. Meanwhile, temporal d Result of proposed deraining algorithm
methods usually bring out blurring artefacts due to high latency. What
is more, if a video combines both rain and moving objects, the situation By solving (1) we obtain the sparse coefficients xsi, i = 1, 2, …, N,
becomes worse. Recently, Kim [1] proposed an optical flow estimation- s = 1, 2. In general, a coefficient with larger absolute value reflects more
based approach, which only needs three successive frames to detect rain important information of an image [4]. An ideal fusion rule is expected
and can also differentiate rain from other moving objects. However, this to enable the fused image to contain the most important information of
method is limited when similar patches in adjacent frames also contain the source images. Therefore, the maximum absolute rule is adopted in
heavy rain streaks. Kang et al. [2] formulated rain removal framework as the sparse coefficient fusion step
an image decomposition problem based on morphological component
analysis (MCA). They decompose the high-frequency (HF) part of an x fi = xŝi , ŝ = arg max { xsi1 },
image into ‘rain component’ and ‘non-rain component’ via sparse rep- s[{1,2}
(2)
resentation. Then, by directly disregarding the ‘rain component’ and i = 1, 2, . . . , N .
using only the ‘non-rain component’ to reconstruct a rain-free image,
the recovered results lose a lot of detail information. Moreover, comput- As illustrated in Fig. 2b, by using only the non-rain bases indicated with
ing the low-frequency (LF) part by averaging consecutive frames creates rectangle in Fig. 2d, Kang’s [3] rain removal method result in the loss of
blurring artefacts in dynamic scenes. detail information. To automatically extend the non-rain dictionary
In this Letter, taking global motions or dynamic objects into consider- without additional heuristically training, we apply dictionary fusion
ation, we first generate an optimally warped frame by estimating the strategy to produce a target dictionary Df by selecting the non-rain
optical flow field. Based on the observation, we find that the warped atoms from dictionaries Ds(s = 1, 2) learned from I1HF and I2HF . In
frame is identical with the current frame except for rain streak other words, we preserve as more non-rain atoms as possible, as
regions. Instead of direct disregarding the rain-affected atoms, we shown in Fig. 2e, with the goal of improving the HF details of the recov-
employ the image fusion strategy to replace rain-affected pixels with ered result in Fig. 2c. Expressed by formula, the updated dictionary
non-rain pixels after classifying the dictionary atoms learned from the Df = [df1, df2, …, dfM] is fused using the following fusion rule:
two frames into the rain cluster and non-rain cluster. In this way, the pro- ⎧
⎪ d1j , d1j [ DN and d2j [ DR ,
posed method is superior to the conventional MCA-based algorithms in ⎪
⎪
both rain streaks removal and details preserving. ⎨ d , d [ D and d [ D ,
1j 1j N 2j N
d fj =
⎪
⎪ d2j , d1j [ DR and d2j [ DN , (3)
⎪
⎩
Preprocessing: From the previous frame Ik−1, the current frame Ik and 0, d1j [ DR and d2j [ DR .
the next frame Ik+1, we generate the optimal warped frame Ĩ k using the
optical flow estimation [1]. The difference between Ik in Fig. 1a and its j = 1, 2, . . . , M .
warped frame Ĩ k in Fig. 1b is illustrated in Fig. 1c, which indicates that
the two frames are identical only except for rain regions. Then, using a
bilateral filter to roughly decompose Ik and Ĩ k into the LF parts I1LF , I2LF
and the HF parts I1HF , I2HF , respectively.
si }i=1 (s = 1, 2) of2 size n × n from Is
N
Patches {yHF HF
are extracted for
learning the dictionary Ds [ Rn ×M via the online-learning algorithm.
Moreover, the optimal coefficient matrix can be obtained by solving


1
x∗si = arg min  yHF si − Ds xsi2 +l  xsi 1
2
a b
2 (1)
i = 1, 2, . . . , N , s = 1, 2.

where N is the number of patches, s ∈ {1, 2} denotes the sth source

image, yHF si represents the ith patch extracted from IsHF and
xsi [ RM ×N are the sparse coefficients of yHF si with respect to Ds. λ is
the regularisation parameter. c d e
To replace the rain-affected components with the non-rain com-
ponents in the fusion step, rain and non-rain atoms should be discrimi- Fig. 2 Deraining results and dictionaries used to reconstruct them
nated first. On the basis of the histograms of oriented gradients (HOG) a Original current frame
b Result of Kang’s [3] method
feature of each atom in the dictionary Ds, we then classify the M atoms in c Result of proposed method
Ds into two clusters: the rain-affected one DR and the non-rain one DN d Dictionary used by Kang
[3, 2]. e Dictionary used by proposed method

ELECTRONICS LETTERS 2nd September 2016 Vol. 52 No. 18 pp. 1528–1529

 From the fused coefficients {x fi }Ni=1 and the fused dictionary Df, we
reconstruct the fused vectors as ŷHF fi = Df x fi , i = 1, 2, . . . , N . After
reshaping each column vector in ŷHF into n × n patch, we set each
f
pixel value in the reconstructed Î HF to be the average of the overlapped
values. Meanwhile, it is observed that rain-affected pixels are brighter
than their original values, we then select pixels with lower value from
the LF parts of the current frame and the warped frame to be the final
LF part. Hence, the recovered rain-free frame is computed as:
Î k = min (I1LF , I2LF ) + Î HF .
a (VIF) [5] between original and derained images. Each PSNR and VIF
Experiments and results: To evaluate the performance of the proposed
algorithm, we compare our method with the most representative method
proposed by Kim [1], Kang’s single-image-based method [3] and video-
based method [2] on various synthetic rain videos that were produced by
‘Adobe After Effect’. In (1), we set the patch size to 16 × 16, the regu-
larisation parameter to λ = 0.15 and the dictionary size to M = 256. For
the subjective evaluation, comparison results on videos with dynamic
scene and static scene are presented in Figs. 3 and 4, respectively. As
can be seen in Figs. 3b and 4b, Kang’s [3] single-image-based video
removal results lose a lot of detail information. Moreover, the video-
based method [2] in Figs.3c and 4c averages temporally adjacent
frames to obtain the LF part of a current frame, which shows severe blur-
ring artefacts due to the dynamic motions or moving objects. In Fig. 3d,
Kim’s [1] algorithm produces a more faithful result, but may cause arte-
facts due to the completion method used. In Figs. 1d, 2c, 3e and 4d, the
effectiveness of the proposed method can be successfully verified.
For the objective evaluation, we measure both the peak
signal-to-noise ratio (PSNR) and the visual information fidelity metric
values are the average over the first 50 frames in the image sequence
used by Fig. 4. High values indicate a better image quality. Table 1
shows that all the considered methods yield satisfactory results, except
for Kang’s single-image-based method [3]. Although the values are
similar with the proposed method, Kang’s video-based result in
Fig. 4c still has some rain streaks remained.

Table 1: Objective evaluation

b c
Kang-single [3] Kang-video [2] Kim [1] Proposed method
PSNR 27.3579 30.9089 34.2739 30.5971
VIF 0.2639 0.5148 0.6228 0.5572

Conclusion: This Letter presents a fusion-based video deraining

method. After classifying atoms in the learned dictionary into rain com-
ponent and non-rain component, the conventional MCA-based method
only uses the remained few non-rain component to recover the rain-free
d e result, which will be a blurred version. From these observations, we
propose a fusion-based method to extend the dictionary used to recon-
Fig. 3 Comparison of deraining results in dynamic scene by struct the result. The proposed method removes rain streaks efficiently
a Original current frame with detail information well preserved, while is suitable for rain
b Kang’s [3] single-image-based method videos with dynamic motions.
c Kang’s [2] video-based method
d Kim’s [1] method
e Proposed method © The Institution of Engineering and Technology 2016
Submitted: 29 April 2016 E-first: 4 August 2016
doi: 10.1049/el.2016.1451
One or more of the Figures in this Letter are available in colour online.
Zetian Mi, Jinxia Shang, Huan Zhou and Minghui Wang (College of
Computer Science, Sichuan University, Chengdu, People’s Republic
of China)
✉ E-mail: wangminghui@scu.edu.cn

References
1 Kim, J.H., Sim, J.Y., and Kim, C.S.: ‘Video deraining and desnowing
a b using temporal correlation and low-rank matrix completion’, Trans.
Image Process., 2015, 24, (9), pp. 2658–2670
2 Kang, L.W., Lin, C.W., Lin, C.T., and Lin, Y.C.: ‘Self-learning-based
rain streak removal for image/video’, Int.Symp. Circuits Syst., 2012,
57, (1), pp. 1871–1874
3 Kang, L.W., Lin, C.W., and Fu, Y.H.: ‘Automatic single-image-based
rain streaks removal via image decomposition’, Trans. Image Process.,
2012, 21, (4), pp. 1742–1755
4 Jiang, Y., and Wang, M.: ‘Image fusion with morphological component
analysis’, Inf. Fusion, 2014, 18, (7), pp. 107–118
5 Sheikh, H.R., and Bovik, A.C.: ‘Image information and visual quality’,
c d Trans. Image Process., 2006, 15, (2), pp. 430–444

Fig. 4 Comparison of deraining results in static scene by

a Original current frame
b Kang’s [3] single-image-based method
c Kang’s [2] video-based method
d Proposed method

ELECTRONICS LETTERS 2nd September 2016 Vol. 52 No. 18 pp. 1528–1529