2014 Middle East Conference on Biomedical Engineering (MECBME)
February 17-20, 2014, Hilton Hotel, Doha, Qatar
Medical Image Fusion Based on Joint Sparse Method
Anuyogam Venkataraman, Javad Alirezaie, Paul Babyn and Alireza Ahmadian
Abstract— In this paper, a novel joint image fusion
algorithm which is the hybrid of Orthogonal Matching Pursuit
(OMP) and Principal Component Analysis (PCA) is proposed
to properly utilize the advantages and to overcome the
disadvantages of both OMP and PCA methods. Firstly,
common and innovative images are extracted from the source
images. Secondly, sparse PCA method is employed to fuse the
information of innovative features. Then weighted average
fusion is used to fuse the sparse PCA result with the common
feature thereby preserving the edge information and high
spatial resolution. We demonstrate this methodology on
medical images from different sources and the experimental
results proves the robustness of the proposed method.
Key words— image fusion, orthogonal matching pursuit,
principal component analysis, sparse representation
I. INTRODUCTION
Multiple images of the same scene can be captured
simultaneously using different sensors. Perceiving the
complete picture of the scene from the captured images of
multiple sensors is not possible. Image fusion algorithms
allow the combination of multiple captured images to
generate a more informative composite image integrating
the complementary information from multiple sensors, even
when they are out of focus and of differing resolution.
Medical Imaging has revolutionized the medical
diagnosis. The arrival of imaging modalities such as
magnetic resonance imaging (MRI) and computed
tomography (CT) tend to give different perspectives of the
same scene which can hinder clinical decision making and
the diagnostic process. For example MRI provides the best
view of soft tissues, while CT is better for its assessment of
bone structures. This motivates the need for image fusion
for precise diagnosis by merging the complementary
information. Three challenges to be addressed while fusing
the images [1] are: 1) The fused image should preserve all
the important information needed for further processing. 2)
Artifacts should not be introduced in fused image. 3) Noise
and unimportant information should be suppressed.
Anuyogam Venkataraman and Javad Alirezaie* are with the
Department of Electrical and Computer Engineering, Ryerson University,
Toronto, ON, M5B2K3, Canada (e-mails: anuyogam1.venkataram@ryers-
on.ca; javad@ryerson.ca, phone: 416-979-5000; fax: 416- 979-5280).
Paul Babyn is with the Department of Medical Imaging, University of
Saskatoon Health Region, Royal University Hospital, Saskatoon, SK, S7N
0W8 Canada (e-mail: paul.babyn@saskatoonhealthregion.ca).
Alireza Ahmadian is with the Department of Biomedical Systems and
Biophysics, Tehran University of Medical Sciences, Poursina Ave.,
Keshavarz Blvd., Tehran, Iran (e-mail: ahmadian@sina.tums.ac.ir, phone:
98-21 66466383; fax: 98-21 66466383).
(Asterisk indicates corresponding author)
978-1-4799-4799-7/14/$31.00 ©2014 IEEE
Sparse representation of signals is now possible utilizing
many different Greedy approaches [3], including: 1.
Matching Pursuit (MP) [3] 2. Orthogonal Matching Pursuit
(OMP) [3], and 3. Stage wise Orthogonal Matching Pursuit
(St OMP) [4]. These techniques are used to represent
signals with the fewest number of non-zero coefficients.
Principal Component Analysis (PCA) [5] is one of the
powerful state-of-the-art image fusion approaches in terms
of visual inspection and quantitative evaluation metrics.
This fusion is carried out by integrating the principal
components of images to be fused. Both PCA and Sparse
fusion have specific advantages and disadvantages. PCA
fusion will enhance the spatial quality but have dense
nonzero entries that might represent uninformative features.
Sparse fusion preserves important information but high
spatial resolution is lacking. This paper proposes a new
algorithm inspired by [6], which employs different fusion
rules for common and innovative sparse components of the
source images. The proposed algorithm utilizes the
advantages of both PCA and Sparse representation for
fusing common and innovative features of the captured
images. This algorithm also overcomes the disadvantages of
both PCA and Sparse representation. In this paper, we
demonstrate the effectiveness of our proposed method by
comparing its results with PCA and Sparse Fusion.
II. METHODS
As discussed in the previous section, sparsest approximation
is achieved through Greedy methods. In this section, we
briefly explore the Orthogonal Matching Pursuit algorithm
to achieve sparsest representation. These sparse coding
algorithms are constructed based on the premise that
Dictionary D of size n×k is already known. For effective
results, we use phase included DCT (Discrete Cosine
Transform) dictionary [7] for our experiment. The signal
𝑆 ∈ ℝ𝑑 is sparse represented 𝑠 ∈ ℝ𝑘 , as given dictionary
𝐷 ∈ ℝ𝑑×𝑘 . In this paper, we have customized OMP sparse
coding algorithm for fusion purposes.
The ultimate aim of OMP algorithm is to achieve best
approximation. The mathematical formula for solving this
constraint problem is given by:
argmin𝑠 ‖𝑆 − 𝐷𝑠‖ 22 ,subject to ‖𝑠‖ 0 ≤ 𝑁 (1)
argmin𝑠 ‖𝑠‖ 0 ,subject to ‖𝑆 − 𝐷𝑠‖ 22 ≤ 𝜖 (2)
Where N is the number of non-zero coefficients. Equation 2
represents the definition for solving error constrained
problem. Next atom to be added in this iterative framework
is the atom which has highest correlation to the residual at
each stage.
103
 Fig.1. Flow of Sparse PCA joint fusion algorithm
Iteration is performed until it meets the stopping criterion.
OMP is due to orthogonalization between atoms in the
dictionary D and residual r.
Pseudo Algorithm of OMP
Given: Dictionary D, signal S, and error threshold ϵ
1) Initialize residual r0=S-Ds0 , index set I0={} and main
iteration is k = k+1 (initial k=0).
2) Using the ideal solution 𝑧𝑖 = 𝑑𝑖𝑇 𝑟 𝑘−1 /‖𝑑𝑖 ‖22 , Calculate
the error 𝑒(𝑖) = min𝑥 �𝑑𝑖 𝑧𝑖 − 𝑟 𝑘−1 � for all i.
3) Update stage: Augmenting the index set 𝐼 𝑘 = 𝐼 𝑘−1 ∪
{𝑖0 } (find i0 of 𝑒(𝑖): ∀1 ≤ 𝑖 ≤ 𝑚 and 𝑒(𝑖0 ) ≤ 𝑒(𝑖).
4) Update the solution 𝑠 𝑘 (𝑖0 )+= 𝑧𝑖 and residual.
5) If stopping criterion is met, 𝑠 = 𝑠 𝑘 ; else, apply another
iteration.
III. PROPOSED SCHEME
This work proposes a fusion method that employs PCA
transformation and sparse transformation. An attempt is
made to efficiently utilize the advantages of PCA and Sparse
fusion scheme. The proposed fusion framework has been
illustrated in Fig.1. Firstly, the common and innovative
components are extracted from geometrically aligned
multiple images of the same scene. Secondly, different
fusion rules are adopted to fuse these coefficients.
The algorithm is summarized as follows:
1) Registered multiple images in an ensemble have one
common component and multiple innovative components.
Innovative components contain the complementary
information from different images of same scene
i structures while sparse fusion comes the second. Result of
2) Innovative components {I i }i =1 of different images i are
decomposed into sparse vectors s1,s2,s3,…si via orthogonal
matching pursuit method described in Figure.1.
104
3) Combine the sparse coefficients of innovative
components using PCA fusion rule, for which the covariance
matrix Cs of innovative images is calculated as follows,
1
𝐶𝑠 = cov(𝐼𝑠 ) = cov([𝑠1 (: ), 𝑠2 (: )])= 𝐼 ∙ 𝐼𝑠 ∗ (3)
𝑖−1 𝑠
Where s1, s2 are the sparse vectors of the innovative
components. Find the Eigen sparse and normalized Eigen
sparse vector of maximum Eigen value. Eigen vector will be
used as weightings for innovative sparse vectors to be fused.
4) Fused PCA result Ip is reshaped into a block of 8×8 and
each pixel position is the sum of several block values.
Reconstructed image I1 is obtained by dividing each pixel
by number of addition operations performed at each pixel.
5) For fusing common component and fused innovative
component, the fuse rule of weighted average scheme
proposed by Burt et al. [2] is adopted.
IV. EXPERIMENTAL RESULTS
In order to test the performance of the proposed joint fusion
algorithm, we compared the quantitative and qualitative
results with two state the of art methods. Qualitative
measurement is done through visual inspection that
considers sharpness and noise suppression. Since the
proposed joint fusion uses both PCA and sparse domain, we
use PCA, Sparse OMP fusion methods for comparison. For
the evaluation, we adopted proposed joint strategy for Multi
resolution, Multi focus and Multimode images and
compared the results with existing algorithms. Our
experiment is carried out with the assumption that the source
images are registered already. Fig. 2, shows results of fusion
for the case of multi focus images. Based on visual
inspection, The Joint PCA Sparse algorithm performs the
best since the resultant image contain more geometric
proposed method seems to contain sharp edges.
 (a) (b)
(c) (d) (e)
Fig.2 Comparison of performance of different fusion
algorithms for case of Multi focus images (512×512) (a)
Focus on right clock (b) Focus on left clock (c) Proposed
Joint fusion (d) PCA fusion (e) Sparse Fusion
(a) (b)
(c) (d) (e)
Fig.3 Comparison of performance of different fusion
algorithms for case of Multi dosage images (512×512) (a)
Low dose Image, PSNR = 22.0994 db (b) Medium Dose
Image, PSNR = 25.1981 db (c) Proposed Joint fusion (d)
PCA fusion (e) Sparse Fusion
Fig.3, illustrates results of fusion of mutli dosage image
results based on three different fusion algorithms. Low dose
image seems to suffer from patchiness. As it can be
observed, visually, Joint Sparse result shows the details
clearly than the Low dose image. Visually, joint fusion
resultant seems to be better than other 2 methods. Increasing
the dosage might reduce the noise but harm patients. Low
dose images are prone to noise. Fusing low dose and
medium dose image should suppress the noise and enhance
the informative details for precise diagnosis.
105
(a) (b)
(c) (d) (e)
Fig.4 Comparison of performance of different fusion
algorithms for case of Multimodal medical images
(512×512) (a) CT Image ((b) MRI Image (c) Proposed Joint
fusion (d) PCA fusion (e) Sparse Fusion
Fig.4, illustrates the results of applying two multi modal
medical images to three image fusion algorithms. The
medical images are MRI and CT image of same scene which
have been registered already. CT image provides the
information on bone structures and MRI image contains
tissue information. Medical image fusion needs great
accuracy as it’s used for diagnosis. Hence, multimodal
image fusion would give sufficient details necessary for
diagnosis. Based on the visual inspection, the Joint Sparse
results contain more detail information. Results of PCA
seem to have high spatial resolution but they are
disappointing in terms of detailed information. Bone details
are not visible in PCA resultant image. Sparse result is better
but some artifacts are easily observed Reconstructing fused
image through joint fusion algorithms seems to be more
precise comparatively.
In order to analyze the quality of the algorithms
quantitatively, we consider 5 metrics: Mutual Information
(MI), PSNR, Correlation, Entropy and Structural Similarity
(SSIM) index. Mutual Information shows how much
information has been transferred from source images to the
resultant images. Entropy shows the amount of important
details available in the image.
PSNR is Peak Signal to Noise Ration which is used to
measure the reconstruction quality of fused image. PSNR of
the fused image If is calculated using the standard formula:
 M2  (4)
PSNR ( I f ) = 10 log10  
 MSE 
Where M is the maximum possible pixel value of the
image and MSE is the Mean Square error. The SSIM [8]
provides structural information of objects and measures the
similarity between the two images.
 TABLE I

PERFORMANCE OF FUSION METHODS BY THE QUALITY EVALUATION METRICS

Image Type Fusion PSNR(db) Mutual Information Entropy Correlation SSIM

Strategy

Multi focus Joint PCA

34.1742 2.1733 7.3656 0.9990 1.000
Fig. 2. Sparse
PCA 31.6321 2.0177 7.2607 0.9981 .9999
OMP fusion 32.3392 2.0606 7.3654 0.9981 1.000
Multi dose
Joint PCA
Fig. 3. 25.2115 0.7887 4.8643 0.9997 1
Sparse
PCA 22.6994 0.7638 4.7905 0.9991 0.9997
OMP fusion 24.4680 0.7794 4.7937 0.9995 0.9998
Multimodal (MRI & Joint PCA
26.4111 0.9634 6.7409 0.9043 .9977
CT) Fig. 4. Sparse
PCA 20.8380 0.8096 6.5502 0.8690 .9919
OMP fusion 24.8056 0.9440 6.7376 0.8985 .9975

Experiment results are shown in Table.1. Tabulated result

demonstrates the effectiveness of the proposed algorithm
over existing methods in terms of Qualitative and
Quantitative methodologies. We can observe that the results
of multimodal image fusion and multi focus image fusion
utilizing our proposed fusion strategy outperforms PCA and
Sparse fusion. The PCA by itself performs poorer results for
all cases.
[2] P. Burt, R. Kolczynski, "Enhanced image capture through
fusion", Proceedings of the 4th International Conference on
Computer Vision, 1993, pp. 173-182.
[3] L. Rebollo‐Neira and D. Lowe, "Optimized orthogonal
matching pursuit approach, "IEEE Signal Processing Letters,
pp.137–140, 2002.

V. CONCLUSION [4] D. Donoho and Y. Tsaig, "Sparse solutions of

Medical Image fusion plays an important role in clinical
diagnosis. In this paper, a joint fusion modal is proposed
based on sparse representation theory and PCA for
multimodal and multi dose medical images. Visually and
quantitatively, the experimental results show that the
proposed method has effectively expressed the geometric
structures and edges and has proved to perform better than
PCA and OMP fusion. This modal can also be extended to
fuse multiple source images from multi resolution, multiple
spectral frequencies and multiple modalities.
REFERENCES
[1] S.G. Nikolov., P.R. Hill., D.R. Bull., C.N. Canagarajah.,
"Wavelets for image fusion", A. Petrosian, F. Meyer (Eds.),
Wavelets in Signal and Image Analysis, Computational
Imaging and Vision Series, Kluwer Academic Publishers,
Dordrecht, The Netherlands (2001). pp. 213–244.
underdetermined linear equations by stagewise orthogonal
matching pursuit," Stanford University, Tech. report 2006.
[5] M. R.Metwalli, A. H. Nasr, O. S. Farag Allah, and S. El-
Rabaie,”Image fusion based on Principal Component Analysis
and High-pass Filter”, Proceedings of the IEEE/ ICCES 2009
international Conference, DEC. 14-16, 2009, pp. 63-70.
[6] H. Yin, S. Li,”Multimodal image fusion with joint sparsity
model”, Opt Eng., 50(6), (2011).
[7] Z. Sadeghipour, M. Babaie-Zadeh, and C. Jutten, "An
adaptive thresholding approach for image denoising using
redundant representations", IEEE international workshop on
Machine Learning for Signal Processing, 2009, pp. 1-6.
[8] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh and Eero P.
Simoncelli, “Image Quality Assessment: From Error Visibility
to Structural Similarity”, IEEE transactions on Image
Processing, vol. 13, no. 4, April 2004.

106