Vous êtes sur la page 1sur 5

DEMOSAICING OF NOISY COLOR IMAGES THROUGH EDGE-PRESERVING

REGULARIZATION

1. Gerace, F. Martinelli

A. Tonazzini*

Universita degli Studi di Perugia

Consiglio Nazionale delle Ricerche

Dipartimento di Matematica e Informatica

Istituto di Scienza e Tecnologie dell'lnformazione

Via Vanvitelli, 1,1-06123 Perugia, Italy

Via G. Moruzzi, 1,1-56124 PISA, Italy

ABSTRACT

We propose edge-preserving regularization for color im


age demosaicing in the realistic case of noisy data. We en
force both intrachannel local smoothness of the intensity, and
interchannel local similarities of the edges. To describe these
local correlations while preserving even the finest image de
tails, we exploit suitable functions of the derivatives of first,
second and third order. The solution of the demosaicing prob
lem is defined as the minimizer of a non-convex energy func
tion, accounting for all these constraints plus a data fidelity
term. Minimization is performed via an iterative determinis
tic algorithm, applied to a family of approximating functions,
each implicitly referring to meaningful discontinuities. Our
method is irrespective of the specific color filter array em
ployed. However, to permit quantitative comparisons with
other published results, we tested it in the case of the Bayer
CFA, and on the Kodak 24-image set.
Index Terms- Color image interpolation, demosaicing,
edge-preserving regularization, non-convex minimization,
color image denoising.

1. INTRODUCTION

Most of the commercial cameras employ a single sensor, as


sociated with a color filter that only permits, at each pixel,
the measurement of the reflectance of the scene at one of the
three Red, Green and Blue colors, according to a predefined
scheme or pattern (color filter array - CFA). This implies that,
for each pixel, the other two missing colors must be estimated,
through color demosaicing techniques. Most of the literature
on demosaicing makes explicit reference to the Bayer pattern
[1], where the green is sampled twice with respect to the red
and the blue, to exploit the higher sensibility of the human
eye to the green wavelength. In [2] a comprehensive survey
of the state-of-the-art can be found.
Partially supported by program POR Calabria FESR

2007-2013 - PIA

Regione Calabria Pacchetti Integrati di Agevolazione Industria Artigianato


Servizi, project ITACA (Innovative Tools for cultural heritage ArChiving and
restorAtion).

978-1-4799-7971-4114/$3l.00 2014 IEEE

A major problem of demosaicing is to avoid oversmooth


ing of the edges. Hence, some methods perform a directional
channel interpolation, after locating the image discontinuities
[3] [4], or analyzing the variance of the color differences [5].
In other methods, the best directional reconstruction of the
missing data is chosen [6], or the two reconstructions are
fused [7]. In particular, [6] proposes an algorithm based on
the Laplacian filter, by selecting the interpolation directions
having the least misguidance level of color artifacts. The Al
ternating Projections algorithm [8] exploits the strong corre
lation between the high frequencies of the three color com
ponents, by alternately projecting the estimated image into an
observation constraint set and a detail constraint set that en
forces similarity between red and green edges and blue and
green edges, until a fixed point is found. The sparse nature
of color images is exploited in [9] to design a suitable dic
tionary applied for demosaicing with the iterative K-SVD al
gorithm. In [lO] total variation is used, while [11] first uses
a quadratic smoothness regularizer and then an adaptive fil
ter, to improve the reconstruction near the edges. In [12] the
missing colors are inferred by exploiting the local geometri
cal image self-similarity, whereas in [13] a level set method
is used to minimize an energy function that gives the direc
tion of the edges along which performing interpolation. For
joint demosaicing and denoising, in [14] the full resolution
green component is first estimated exploiting both spectral
and spatial correlations to suppress sensor and interpolation
errors, and then the CFA-dependent noise is removed using
wavelets. In [15] the luminance and chrominance channels
are first reconstructed by exploiting a frequency analysis of
the Bayer pattern. Wiener filters are then designed to denoise
the chrominances, whereas the luminance is linearly filtered.
The work [16] proposes a modified total least squared estima
tion, to derive a demosaicing filter able to deal with the noise
affecting the base vectors. In [17], the authors evaluate the
statistical characteristics of the noise resulting from their pre
vious demosaicing method [11], and then remove it through
post-processing.
In this paper, we propose regularization for joint demo
saicing and denoising, considering both intrachannel and in-

terchannel local correlation constraints. We assume the in


tensity of each channel to be locally regular, and enforce this
constraint through stabilizers that implicitly address intensity
discontinuities of first, second and third order, respectively.
This allows to reconstruct very complex scenes with fine de
tails while removing noise. Interchannel correlation is en
forced in correspondence of the image high frequency com
ponents, through stabilizers that promote the amplitude of
the intensity discontinuities in the different channels to be
equal almost everywhere, except than on a set of implicit
hyper-discontinuities. The above constraints plus the data
fidelity term are merged into a non-convex energy function,
whose minimizer is estimated via an iterative deterministic
algorithm, entailing the minimization in sequence of a family
of approximating functions that, starting with a first convex
one, gradually converges to the original energy.
The paper is organized as follows. In Sections 2, the
edge-preserving regularization model adopted is introduced.
In Section 3, the solution strategy is detailed. Section 4 is de
voted to the quantitative comparison with some of the most
performing algorithms of the recent literature, and, finally,
conclusions and future prospects are presented in Section 5.
2.

THE EDGE-PRESERVING REGULARIZATION


MODEL

N
gk,N

gk,N

IIM(x - y)ll; + L Ai: L gk,N (N:x) +


k=l CECk
(1)
+ L Ak L gk,v (Vkc X) ,
k=1 CECk
3

where 1111 indicates the Euclidean norm, and the first term
expresses a data fidelity term, which is effective when the
data are noisy and identically zero in the noiseless case. The
second term is the sum of stabilizers expressing local intra
channel correlation, the third term is the sum of stabilizers
expressing local interchannel correlation, and
and
are
positive parameters balancing the relative weight of the three
constraints. The terms
are the norms of the vectors of the
R, G and B intensity derivatives of a certain order k:

Ai:

Ak

Vkc X=
II (D xR - DxG, DxR - DxB, DxG - DxB) II (3)
In this way, the amplitudes of the channel intensity discon
tinuities are intended to share the same hyper-edges. Analo
gously to the case of
and with the same meaning, also
operator
is weighted by a suitable neighbor interaction
function, to permit the smoothness constraint to be local. We
adopt neighbor interaction functions
and
all having
the same functional form of the truncated parabola [18], but
containing a parameter
that can instead vary from one to
another:
if
<
(4)
otherwise

N,

gk,N

Ck

gk,v

/-1,

g(t) =

{ t/-1,22

ItI

/-1,

3. SOLUTION S TRATEGY
The solution to our problem is given by the global minimizer,
with respect to
of the non-convex en
ergy function of eq. (1). Our solution strategy is inspired
to the Graduated Non-Convexity (GNC) algorithm [18], and
consists in defining a family of approximations
P
PI, ... ,Po, PI > 0, Po
0, for the original non-convex en
ergy function, where the first approximation
is convex
and the successive ones gradually converge to the original one
The various approximations are minimized in
sequence, using the previous minimizer as starting point for
the subsequent minimization. If the first convex function is
chosen to be not too far from the original energy, and all the
other approximations are close to each other, the convergence
to the global minimum of the original, non-convex function
is empirically ensured by the fact that the starting point for
minimizing
should be located in the valley of its global
minimizer. The original version of the GNC algorithm was

x = (xR,xG,xB),
=

E(O) = E.

where
is the finite difference operator computing, for
each pixel of a given color channel, the numerical partial
derivative of order k over a suitable set c of adjacent pix
els. Thus
represents the union, across the image grid,
of the all the sets c involved in the numerical derivatives of
order k. To preserve even the finest details in the images,
we consider intensity derivatives of first, second and third
order, respectively, that is k
1, 2, 3. Operator
returns,

Vkc ,

Vkc

The energy function we consider is:

E(x)

at each pixel and each partial derivative, a cumulative gra


dient measure for the three R, G and B channels together,
which means that they all share the same edges. Each
is weighted by a suitable neighbor interaction function
that makes the cumulative gradient to decrease when it is al
ready small, thus inhibiting intensity discontinuities due, e.g.,
to noise, and promoting a smooth filling-in of the lacking
pixels. Conversely,
makes
to decrease less when it
is greater than a certain threshold, thus preserving existing,
truly intensity discontinuities. Nevertheless, the correspond
ing intensity discontinuities at the single channels might be of
different amplitude. This is not desirable if a a constraint of
interchannel correlation has to be enforced, since it is know
that cross-correlation mainly occurs in correspondence of the
image boundaries, textures and details. Thus, the amplitude
of the intensity discontinuities of the color channels should
be similar almost everywhere. To this purpose, in a recursive
manner, this similarity is measured by the terms
which
are the norms of the vectors of the three interchannel first
order finite differences of the intrachannel derivatives:

E(O)

E(p), =

E(Pl)

derived for the case of denoising only, that is for a data term
strictly convex, whose Hessian is a positive definite matrix.
Thus, building a first convex approximation only required a
mild "correction " of the concavity of the neighbor interaction
functions. In our case, we have the further difficulty that the
our data term is semi-definite positive, being M a singular
matrix, so that making convex the overall energy function re
quires the adoption of stronger "corrections " for the concavity
of the neighbor interaction function.
In general, the edge-preserving property is only requested
for the original, non-convex energy function. However, when
generic approximations are used, to obtain a solution satis
fying the desired constraints, all the steps of the GNC must
be executed, and p = 0 must be reached. Conversely, our
aim is to build a family of approximations that are all edge
preserving, in such a way that, even stopping the GNC al
gorithm at some p > 0 to reduce the computational burden,
we are guaranteed that the solution has correct discontinu
ities. Furthermore, when p > 0 the edge elements of the
reconstructed image are continuous-valued in
rather
than boolean as when p = O. Accounting for "soft " edges
is often preferable than having "hard " edges, since oblique or
curvilinear object borders can be reconstructed in a more nat
ural and visually pleasant way. Using boolean edges, these
borders will be instead depicted as sharp horizontal and ver
tical lines, thus producing an annoying "zig-zag " effect, the
so-called aliasing effect.

(0,1),

4. EXPERIMENTAL RESULTS

The algorithm proposed in this paper has been tested for the
Bayer CFA on the set of the 24 high quality row Kodak sam
ple images [21], which represent the typical benchmark im
ages used in the literature to compare the different demosaic
ing algorithms. The free parameters appearing in the energy
function have been calibrated on these images. More pre
cisely, through a trial-and-error strategy, driven by the expe
rience and of course non exhaustive of all possible combina
tions, we looked for the set of parameters that give the best
average Peak Signal-to-Noise Ratio (PSNR) on all the 24 im
ages. The algorithm has been applied to both noiseless and
noisy images. In the noiseless case, we compared the pro
posed method with some of the most performing methods in
the literature, namely the algorithms proposed in [8], [6], [9],
[11], [3], [12], and [13], respectively. In the noisy case, we
compared our method with the algorithms in [8] and [16], by
using the original MATLAB code provided by the authors,
and with the algorithms in [15], [14], and [17], by using the
source codes available in the authors's web pages.
In the noiseless case, the best free parameters have been
empirically found to be the ones reported in Table 1.

Table 1.

AN

""N

In our family of approximations for the energy of eq. (1)

we substitute the original stabilizers with functions g kP and


gkP , p= 2,... , 0, having a functional form dependin on p,
O
O
an including parameter"" as well. For p=0, gk and g k
must coincide with the neighbor interaction funtions of q.
(4). For p= the first approximation of E must be convex.

2,

k2 and gk2 , the following

This is achieved by using, as g


"
function [19]:

(2) ()t
9

where

- { t2
_

2qltl_ q 2

It I < q

otherwise

(5)

is a positive arbitrary

constant, and
represents the proper regularization param
eter, out of the sets of the
s and
s appearing in the
energy. The different approximations adopted for values of p
in the ranges
and
are specified in [20]. Note that,
whereas when p = 0 the hidden edge elements are boolean,
at each p > 0 they are continuous-valued with values be
tween 0 and
leading to the advantages mentioned above.
The minimization of each approximated energy is performed
by means of a standard descent technique, namely the Non
Linear Successive Over-Relaxation (NL-SOR) [18] iterative
algorithm.

(0,1]
1,

(1,2)

""v

k=2
0.04
8.66
0.078

k=3
0.04
7.07
0.078

3.46

6. 9

6. 9

Ak

As per the algorithm, we chose to decrease p with a step of


0.01; for each sample image and for each value of p, we com
puted the R MS E between the ideal image and the minimizer
of the approximated energy function, indicated as T)j (p), j =
... ,24. The value p to which stop the algorithm has been
determined as:

p=arg min
P

Af

AV

k=l
0.0 1
30
0.25

1,

,\1 ) -2, T*
T*

( 2

vA

if

Parameters used for the noiseless case

{f (P)} .
j=l

T)j

(6)

For the given free parameters, we found p = 1.40. For each


test image, the reconstruction obtained with this value of p
has then be taken as the optimal reconstruction provided by
our algorithm.
By looking at Table 2, it is apparent that the results ob
tained with the proposed method exhibit a PSNR higher than
that of the results obtained with the other methods in the 54%
of the cases (for each image, the higher PSNR is highlighted
in boldface). Figure 4 shows a portion ofImage 3, where our
results are compared with those obtained with the algorithms
in [8].

Fig. 1.

lm*lge
I

9
10
II
12
13
14
15
16
17
18
19
20
21
22
23
24

From left to right: original detail of Image 3, result of the proposed method and result of the algorithm in [8].

Table 2. PSNRs for noiseless images

[S[
37.7
39.57
41.45
40.03
37.46
38.5
41.77
35.08
41.72
42.02
39.14
42.51
34.3
35.6
39.35
41.76
41.11
37.45
39.46
40.66
38.66
37.55
41.88
34.78

[6]
35.17
39.34
41.52
38.87
35.7
37.55
40.87
33.8
41.1
40.77
37.48
41.81
31.41
35.5
38.02
41.37
39.25
35.2
38.44
39.23
36.56
36.46
41.88
33.42

[9]
39.37
40.71
43.19
41.29
38.7
40.05
42.83
36.42
43.28
42.7
40.22
43.53
35.29
37.95
40.21
43.62
42.01
37.47
41.27
41
39.74
38.87
42.41
35.63

[II[
38.22
38.18
42.(14
40J14
38.04
39.7
42.1
36.08
42.15
42.15
39.78
42.94
34.94
36.34
39.15
43.27
41.83
37.13
40.15
40.39
39.27
38.25
40.4
35.37

[13]
35.64
36.46
37.25
36.74
35.45
36.39
37.07
34.59
37.46
37.26
36.41
37.56
33.68
35.07
36.22
37.53
41.09
35.98
40.20
32.49
36.47
37.32
39.45
34.32

[12[
32.61
32.51
32.42
32.48
32.63
32.52
32.44
32.74
32.40
32.42
32.52
32.38
32.86
32.68
32.54
32.39
31.99
32.57
32.09
33.01
32.51
32.41
32.17
32.78

[3]
39.96
40.)
43.26
40.56
38.31
41
42.64
37.35
43.42
42.83
4o.r.6
44.13
36.03
37.1
39.84
44.47
41.77
37.96
41.79
41.71
39.99
38.48
43.2
35.39

Table 4. PSNRs for noisy images, a=8.

im*lge
I

proposed
40.66
40.86
43.31
41.84
38.94
40.20
43.62
37.22
43.29
42.70
40.51
44.40
36.24
38.26
40.35
43.75
41.54
37.43
41.10
41.59
40.20
38.48
43.89
34.78

10
II
12
13
14
15
16
17
18
19
20
21
22
23
24

For the noisy case, we considered images corrupted by


independent, Gaussian noise, with zero mean and several dif
ferent values of the standard deviation a. The best free pa
rameters for all the 24 images have been empirically found to
be dependent on a according to Table 3.
Table 3. Parameters used for the noisy case
AN
",N

AV
",v

k=l
O.la

k=2
0.05a

k=3
0.05a

5
5
f!
f!
f!
la
0.05a
0.05a
O.

f! 5 f! 5 f!

For each value of the noise standard deviation, the suitable


value p for stopping the algorithm has been found. The fol
lowing empirical law that relates p to a has been determined:

3
4
p(a) = -a +-.
5
40

(7)

The PSNR values computed for the case a = 8 are shown in


Table 4. From the Table, it is apparent that the PSNRs of the
reconstructions obtained with our algorithm are largely higher
than those of the other algorithms.

bilinear
23.38
25.86
25.98
25.84
23.68
24.09
25.78
21.89
25.57
25.58
24.78
25.69
22.03
24.76
25.64
25.31
25.70
24.29
24.30
26.00
24.43
25.13
26.07
26.98

[S]
24.24
24.50
24.47
24.38
24.42
24.40
24.43
24.18
24.36
24.38
24.45
24.44
24.12
24.25
24.79
24.36
24.68
24.39
24.35
25.44
24.32
24.28
24.47
25.26

[16[
22.35
23.55
23.84
23.48
22.85
23.09
23.47
22.18
23.66
22.85
23.28
23.72
21.99
22.96
23.94
23.44
23.79
23.01
22.93
24.80
23.21
23.16
23.96
25.50

[15]
31.64
30.16
35.56
32.80
31.98
32.58
34.74
31.17
35.80
35.49
33.19
34.72
30.03
31.81
32.%
34.49
35.08
30.90
33.26
34.04
32.65
37.17
33.39
29.73

[14[
31.49
30.17
34.91
32.56
31.29
32.39
34.01
30.74
35.13
34.57
32.85
34.37
30.03
31.55
32.72
34.16
34.39
30.69
32.87
33.62
32.36
32.18
33.24
29.71

[17]
28.89
31.72
33.65
32.02
29.70
30.21
33.57
28.92
33.61
33.35
30.91
33.01
27.53
29.51
32.12
31.71
32.69
29.68
31.00
32.33
30.91
30.55
34.84
27.32

proposed
31.76
34.17
35.74
34.29
31.90
32.49
3538
31.00
35.45
35.15
33.03
34.39
29.75
32.15
34.11
34.12
34.56
31.70
33.26
34.38
33.27
32.78
35.01
29.74

5. CONCLUSION

We approached the joint demosaicing and donoising of color


images within a regularization approach, which is irrespec
tive of the CFA employed to generate the data. A main fea
ture of our method is the adoption of local image smoothness
models that implicitly account for continuous-valued edges
both at the low and high frequency level. Graduated edges
are important for preventing the aliasing effect and for pre
serving the fine details in the image, especially when noise is
present. This is achieved in two ways: on one hand, we in
clude derivatives up to the third degree; on another hand, we
weight these derivatives through non-quadratic neighbor in
teracting functions. An exhaustive experimentation over the
Kodak 24-image dataset under the Bayer CFA, in different
conditions of noise, demonstrates the good performance of
our method, against some of the best performing demosaic
ing algorithms proposed so far. Planned future developments
regard experimentation of the algorithm on data from other
CFAs, and the inclusion of the filter blurs that, in this kind
of application, are known. As per this latter issue, the exten
sion of the algorithm is straightforward, being the data term
still convex, so that the same approximations for the energy
function could be exploited.

6. REFERENCES

[1] B.E. Bayer,


"Color imaging array, " U.S. Patent
3971065, Jul. 1976.
[2] Daniele Menon and Giancarlo Calvagno, "Color image
demosaicking an overview, " Signal Processing: Im
age Communication, vol. 26, no. 8-9, pp. 518-533, Oct.
2011.
"Low[3] King-Hong Chung and Yuk-Hee Chan,
complexity color demosaicing algorithm based on inte
grated gradients, " 1. Electron. Imaging, vol. 19, no. 2,
2010.
[4] A Hore and D Ziou, "An edge-sensing generic de
mosaicing algorithm with application to image resam
piing, " vol. 20, no. 11, pp. 3136-3150, 2011.
[5] King-Hong Chung and Yuk-Hee Chan, "Color demo
saicing using variance of color differences, " vol. 15, no.
10, pp. 2944-2955, 2006.
[6] K. Hirakawa and T. W. Parks, "Adaptive homogeneity
directed demosaicing algorithm, " vol. 14, no. 3, pp.
360-369, 2005.
[7] D. Menon and G. Calvagno, "Demosaicing based on
wavelet analysis of the luminance component, " in Im
age Processing (ICIP), 2007 14th IEEE international
Conference on,

2007, vol. 2.

[8] B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau,


"Color plane interpolation using alternating projec
tions, " vol. 11, no. 9, pp. 997-1013, Sep. 2002.
[9] J. Mairal, M. Elad, and G. Sapiro, "Sparse represen
tation for color image restoration, " vol. 17, no. 1, pp.
53-69, 2008.
[10] T. Saito and T. Komatsu, "Demosaicing approach based
on extended color total-variation regularization, " in Im
age Processing (lCIP), 2008 15th IEEE international
Conference on,

2008, pp. 885-888.

[11] D. Menon and G. Calvagno, "Regularization approaches


to demosaicking, " vol. 18, pp. 2209-2220, 2009.
[12] A Buades, B Coll, J.M Morel, and C Sbert, "Self
similarity driven color demosaicking, " vol. 18, no. 6,
pp. 1192-1202, 2009.
[13] S Ferradans, M Bertalmfo, and V Caselles, "Geometry
based demosaicking, " vol. 18, no. 3, pp. 665--670, 2009.
[14] L Zhang, X Wu, and D Zhang, "Color reproduction
from noisy CFA data of single sensor digital cameras, "
vol. 16, no.9, pp. 2184-2197, 2007.

[15] L Condat, "A simple, fast and efficient approach to de


noisaicking: Joint demosaicking and denoising, " in Im
age Processing (ICIP), 2010 17th IEEE International
Conference on,

2010, vol. 50, p. 1.

[16] K. Hirakawa and T.w. Parks, "Joint demosaicing and


denoising, " vol. 15, no. 8, pp. 2146-2157, Aug. 2006.
[17] D Menon and G Calvagno, "Joint demosaicking and de
noisingwith space-varying filters, " in Image Processing
(ICIP), 2009 16th IEEE International Conference on,

2009, pp. 477-480.


[18] A. Blake and A. Zissermann, Visual Reconstruction,
MI T Press, Cambridge, MA, 1987.
[19] D. Shulman and J-Y. Herve, "Regularization of dis
continuous flow fields, " in Visual Motion, 1989,Proc.
Workshop on, Mar. 1989, pp. 81 -86.
[20] Antonio Boccuto, Ivan Gerace, and Patrizia Pucci,
"Convex approximation technique for interacting line
elements deblurring: a new approach, " Journal of Math
ematical Imaging and Vision, Online First, 27 Septem
ber 2011.
[21] "Kodak eastman company, "photocd pcd0992 ",, " .