Gaussian Mixture Model Based Scheme Using Expectation-Maximization for Image Segmentation

Ritu Garg (2011EEZ8469) Adersh Miglani (2011EEZ8471) Project - EEL709 Course IIT Delhi 27-April-2012

AbstractIn this paper, we dene a probabilistic model where each class is represented by nineteen dimensional multivariate distribution over 3 3 regions in image. This distribution is used to model classes are Gaussian mixture models learned from the training data using the Expectation Maximization (EM)algorithm. For the experiments we have used the UCI image segmentation database.

In section IV we present and discuss our experimental results and nally conclude in section V.

II. DATA S ET P REPROCESSING We have used the UCI image segmentation dataset1 to evaluate EM based GMM learning. We summarize the relevant information related to the dataset below:

I. I NTRODUCTION Image segmentation is useful in many applications. Goal of image segmentation is to partition an image into regions each of which has a reasonably homogeneous visual appearance or which corresponds to objects or parts of objects. The image segmentation is in general extremely difcult and remains the subject of active research. Here we implement a variate of GMM using EM based technique to classify pixels in the image segment objects in the image. Researchers have employed the theory of Gaussian mixture models for image segmentation [1][3]. The images can be considered as a mixture of multi-variant densities and the mixture parameters are estimated using the EM algorithm. The segmentation is completed by classifying each pixel into its corresponding class according to the maximum log-likelihood estimation. A major drawback of this method is that the number of Gaussian mixture components is assumed to be known prior, so it cannot be considered as a complete unsupervised learning methodology. Another issue is the problem of mixture parameter initialization during EM algorithm that can greatly effect the performance. A commonly used solution is initialization by randomly sampling in the mixture data [4] [5]. But this method may result in inconsistent mathematical models such as non-linear convergence, inconsistent covariance matrix etc. In our experiments, we have used K-means algorithm to initialize the Gaussian mixture parameters and successfully solve the initialization problem. Hence we can state that method performs both parameter estimation and model selection in a single algorithm, thus the method is totally unsupervised. The paper is organized as follows: In section II we describe the various pre-processing steps applied to the give dataset to rule out the inconsistency in the dataset. Further in section III we discuss the Gaussian mixture model and EM algorithm.

Each instance is a 3x3 region. Total number of Instances = 2310, 210 belonging to test data and rest Number of Attributes: 19 continuous attributes. Missing Attribute Values: None Class Distribution: 1 2 3 4 5 6 7 = = = = = = = brickface, sky, foliage, cement, window, path, grass.

At rst we started with considering all the training samples (i.e. 210 19 data). It was found that the covariance matrix computed using this data had one column and one row as zero. This implies that, that particular dimension was same for all samples. This was detected using the variance across each sample for every dimension and redundant data is removed from the feature space. Further, we applied SVD on the reduced feature set and noticed that the eigen values for few to the dimension was extremely small ranging from 1015 1025 also some of eigen values computed were negative. In order to make covariance matrix consistent and positive denite we applied Principal Component Analysis (PCA). Principal component analysis was to transform original mean-adjusted features into new eigen space with dominant dimensions, resulting in a consistent positive denite covariance matrix for each iteration.
1 http://archive.ics.uci.edu/ml/machine-learning-databases/statlog/segment/

III. G AUSSIAN M IXTURE M ODELS AND EM A LGORITHM A. Gaussian Mixture Models Gaussian mixture model is a simple linear superposition of Gaussian components, aimed at providing a richer class of density models than the single Gaussian. The Gaussian mixture model with K components for xn samples, n 1, can be written as :
K

p(x) =
k=1

k N (x|k , k )

(1)

Where k is mixing coefcient, k and k are the set of parameters dening the K th component to be learned using EM algorithm. N (x|k , k ) is the Gaussian distribution with mean k and covariance k . B. EM Algorithm The EM algorithm is a general iterative technique for computing maximum-likelihood. The usual EM algorithm consists of an E-step and an M-step. Given a Gaussian Mixture, the goal is to maximize the log-likelihood function with respect to the parameter vector comprising means and covariances of components and mixing coefcients. The EM-algorithm is summarized below: Initialize means k , covariances k and mixing coefcient k based on standard K-means clustering algorithm. E-Step: Evaluate the responsibility using current parameters. (znk ) =

IV. E XPERIMENTAL R ESULTS As part of pre-processing i.e. after removing redundant dimension and applying PCA, resulted in a reduced dimension data of size (210 12) which was used for subsequent processing and learning. We used K-means algorithm on the new sample set that yielded acceptable means, covariances. Undersigned learning using EM based GMM provides local minima under constraint that initialization of accepted Gaussian parameters. Since this algorithm can get trapped into one of many local maxima of likelihood function. Therefore we adopted multi-modal Gaussian for each class. For example in each of seven classes we t K = 2, 3, 4 Gaussian. The convergence criterion of learning Gaussian parameters in EM step was change of 1e2 . For every sample we compute p(x) as sum of Gaussian over all K classes. This is achieved by maximum log-likelihood as objective function which is the sum of Gaussian. If any sample is equal to the mean, log-likelihood function will tend to innity. Thus maximization of log-likelihood of regular Gaussian is not a well posed problem. Due to above mentioned singularity. An elegant and powerful method for nding maximum likelihood for models with latent variable k is EM algorithm.

k N (xn |k , k ) K j=1 j N (xn |j , j )

(2)

M Step: Re-estimate the parameters using current responsibilities. new = k

K

1 Nk

(znk )xn
k=1

(3)

new = k

1 Nk

(znk )xn (xn new )(xn new )T k k

k=1

(4)
new k =

Nk N
K

(5)

where

Nk =
k=1

(znk ).

(6)

Evaluate the log-likelihood

N K

lnp(X|, , ) =
n=1

ln{
k=1

k N (x|k , k )}

(7)

Fig. 1.

Threshold vs. Classication over Seven Classes

Since, EM algorithm is highly dependent on initialization, instead of performing random sampling we have used K-means to initialize the mixture parameters.

The 7-class problem is experimented with varying the number of components K = 2, 3, 4, 5 in each class. Figure 1 illustrates the allocation of test samples in K = 7 classes with varying KInClass = 2, 3, 4, and 5.

Fig. 2. Threshold vs. Expected Number of Samples over Total 2100 Samples Fig. 3. Negative log-likelihood with experiments

We verify our result with UCI test image database. As shown in the second row of Figure 1, expected number of samples over total 2100 samples are correctly classied based on UCI test-set. Figure 2 illustrates change in expected classied test samples over varying negative likelihood threshold, that is change in likelihood in every iteration with respect to previous one. Figure 3 shows the plot for negative log-likelihood vs. number of iterations consumed to converge while estimating mean, variance and mixing coefcient for each Gaussian. It is clear that our implementation shows consistent result for multi-modal K Gaussian tting in each class with monotonically decreasing negative log-likelihood. The expected number of samples with respect to the classes specied by UCI for each test sample is very low. We performed following steps to chose dominant dimensions from ill-conditioned UCI database and same may lead to approximation in our results. We compared variance of each dimension over each samples and found three classes are redundant as per assumed minimum variance as 0.001. The variance for other dimensions was up to in the multiple of 100. We performed PCA and compared all 19 eigen values. Out of sorted list of eigen values, highest eigen values was in multiple of 1000 and last four eigen values were below 1013 . Such lower eigen values should be considered as zero. All training and test mean-adjusted samples were transformed in new eigen space with 12 dimensions V. C ONCLUSION In this paper, we present an unsupervised image segmentation method based on nite Gaussian mixture model. The observed pixels of the 3 3 region was considered as a mixture of multi-variate densities. Each 3 3 entry is represented using 19- dimensional features. In our experiments, it was essential to perform pre-processing on the data to avoid inconsistent mathematical modelling. Further, we used the Kmeans algorithm that successfully circumvent the initialization problem of EM algorithm. Finally, we are able to show consistent convergence for association of test samples with

their respective clusters based on the three variate parameters learned using EM-GMM. Same is indicated in gure 3. R EFERENCES
[1] T. Yamazaki and T. Yamazaki, Introduction of em algorithm into color image segmentation, in Proc. ICIPS98, 1998, pp. 368371. [2] H. Caillol, W. Pieczynski, and A. Hillion, Estimation of fuzzy gaussian mixture and unsupervised statistical image segmentation, Image Processing, IEEE Transactions on, vol. 6, no. 3, pp. 425 440, March 1997. [3] J. J. veeberk N. Vlassis and B. Klose. [4] G. McLachlan and T. Krishnan, The EM Algorithm and Extensions. Wiley, 1996. [5] G. McLachlan and D. Peel, Finite Mixture Models. Wiley, 2000.