Robust Moving Object Segmentation Approach based

on GMM and Optical Flow

Rahul Paul

Dr. Sushanta Mukhopadhyay

M.Tech Computer Science & Engineering Department

ISM Dhanbad
Dhanbad,India
hellorp1990@gmail.com

Asso. Prof Computer Science & Engineering Department

ISM Dhanbad
Dhanbad,India
msushanta2001@yahoo.com

Abstract Moving object segmentation and tracking are two

essential steps for surveillance videos, capturing human motion
etc. There are three conventional approaches to moving object
segmentation and detection: Background Subtraction, Optical
Flow and Temporal Differencing. Gaussian mixture model
(GMM) based background subtraction has been widely employed
in image processing in recent years. But the disadvantage of
GMM algorithm is that it cant deal with sudden lighting changes
and initializing the Gaussian is needed. Optical Flow is apparent
motion field of the captured scene in the image so that we can
distinguish the moving objects. In this paper we propose to
incorporate Optical Flow information with the Gaussian Mixture
Model and Markov Random Field (MRF). First, GMM and
optical flow methods are used separately to segment the moving
object and results obtained from both methods are combined
together. Finally, MRF is used to refine the segmented objects to
obtain a reliable result. Experimental results show that the
proposed method is better than the traditional GMM method.
Keywords Background Subtraction; GMM; Optical Flow;
Markov Random Field

I.

Introduction

Moving foreground object detection, segmentation and

tracking in a video sequence is a fundamental step in a wide
range of computer vision application including medical
imaging, traffic monitoring, and video understanding. Moving
object detection involves identification of an object in
consecutive frames until it disappears. Moving objects contain
critical information and features in surveillance videos, the
detection and segmentation of moving objects provide input to
important vision task, such as object classification , behavior
recognition and intrusion analysis. In this context there are
many issues, such as reflections, shadows, dynamic
foreground, illumination change etc , that make the obtaining
high accuracy segmentation and tracking very difficult and
error prone.
To tackle these unwanted problems, intensive research has
been conducted. An important assumption in many other as
well as in our approach is that the camera remains static while
capturing the video. Background subtraction is one of the most

popular methods for detect and segment the foreground

objects in video streams .Background subtraction is performed
by subtracting the current frame from the background image
and generates a foreground mask for every frame. Gaussian
Mixture Model (GMM) is one of the most commonly used
background subtraction model due to its capability to adapt to
background variations. It is proposed by Friedman and Russell
.GMM is a single extension of the Gaussian probability
function which can give a compact representation for video
object or sub-object regions and the model is propagated
frame by frame. As the GMMs can approximate any smooth
shape of the density distribution, so often used in image
processing in recent years. However this method is susceptible
to illumination changes .Another disadvantage of GMM
algorithm is that the user must set the number of mixture
models that the algorithm will try and fit to the training
dataset.
Uniform motion of moving object can be calculated by
Optical Flow algorithms. The concept of optical flow was
introduced by the American psychologist James J Gibson in
the 1940s to describe the visual stimulus provided to animals
moving through the world. It is a 2D vector field where each
vector is a displacement vector showing the movement of
points from one frame to another. Optical flow is used by
robotics researchers in many areas such as: object detection
and tracking, image dominant plane extraction, movement
detection, and robot navigation. Optical flow can detect the
moving object even when the camera moves, but it needs time
for its computational complexity and optical flow estimation
only involves local computation. To solve the ambiguity left
by GMM and Optical Flow methods, we propose to
incorporate the optical flow model into the GMM method.
Another important aspect of this paper is MRF based
refinement of the initial segmentation result to generate a
reliable video object segmentation result. A Markov random
field is a set of random variables having a Markov
property described by an undirected graph. MRF theory tells
us how to model the a priori probability of contextual
dependent patterns, such as a class of textures and an
arrangement of object features. Problems in computer vision

usually involve noise, and so exact solutions are most often

impossible. Additionally, the latent variables of interest often
have the Markov property. For example, the pixel values in an
image usually depend most strongly on those in the immediate
vicinity, and have only weak correlations with those further
away. Markov random field (MRF) theory provides a
convenient and consistent way of modeling context dependent
entities such as image pixels and other spatially correlated
features. Therefore, vision problems are well suited to the
MRF optimization technique.
The rest of the paper is organized as follows: Section II
reviews some relevant works on foreground object
segmentation. Section III describes our motivation for the
proposed work and our proposed algorithm Experimental
results are shown in section IV and after that a brief
conclusion in section V.

II.

Literature Review

The detection of interesting foreground object from a video

sequence provides a classification of the pixels into either
foreground or background. When the scene is stationary or
gradually evolving, the foreground object detection can be
done using many traditional background subtraction
algorithm . One of the important work in the frame
differencing method has done by Jain and Nagel[1979] .
However background subtraction became popular following
the work of Wren et al. [1997]. Elgammal and Davis [2000]
use nonparametric kernel density estimation to model the
background . Li and Leung[2002] combine the texture and
color features to perform the background subtraction.
Another interesting approach for background subtraction is
Hidden Markov Model(HMM) . Stenger et al. [2001] use
HMM for background subtraction. One of the famous
temporal modeling method known as Mixture of Gaussians
(MoG) proposed by Stauffer and Grimson [1]. In this
method a pixel in the current frame is checked against the
background model by comparing it with every Gaussian in
the model until a match is found. For image segmentation
and clustering approach Comaniciu and Meer[2002]
propose the mean shift approach to find the clusters in the
joint spatial and color space. Almost all work in image
sequence processing begins by attempting to find the vector
field which describes how the image is changing with time.
Ideally, the projection into the two dimensional image plane
of the three dimensional velocity field seen by the camera
should be computed. Horn and Schunk in [2] give the
example of a rotating sphere with no surface markings
which, under constant illumination, causes no changes in
the image intensity over time .
This paper takes a useful and simple approach to detect and
segment moving object and try to remove the shortcomings
of GMM and Optical flow. The result shows that our
proposed method improves the segmentation method.

Proposed Moving Object

Segmentation Approach

III.

A. Gaussian Mixture Model

A Gaussian Mixture Model (GMM) is a parametric
probability density function represented as a weighted sum of
Gaussian component densities. GMMs are commonly used as
a parametric model of the probability distribution of
continuous measurements or features in a biometric system,
such as colour based tracking of an object in video.
Pixel values that do not fit the background distributions are
considered foreground .This is a common method for real time
segmentation of moving regions in image sequences. Model
Gaussians are updated using K-means approximation method .
Each Gaussian distribution is assigned to represent the
background or moving a moving object in the adaptive
mixture model. Every pixel is then evaluated and classified as
moving region or as a background.
In order to sample from a GMM, one samples first the
component index k{1,,K} with prior probability k, and
then samples the vector xRd from the k- th Gaussian
distribution p(x|k,k). Here k and k are respectively
the mean and covariance of the distribution. The GMM is
completely specified by the parameters ={k,k,k;k=1,
,K}.
The density p(x|) induced on the training data is obtained by
marginalizing the component selector k, obtaining
(1)
(2)
Learning a GMM to fit a dataset X=(x1,, xn) is usually
done by maximizing the log-likelihood of the data:
A Gaussian mixture model can be formulated in general as
follows:
(3)

Where obviously,

. The mean of such a mixture

equals
that is the weighted sum of the
means of the component densities. Where, K is the number of
distributions and t represents time,
is an estimate of the
weight of the i th Gaussian in the mixture at time t.
B. Optical Flow
Optical flow is defined as the apparent motion of image
brightness patterns in an image sequence. The optical flow is
a vector flow field that records point wise, the instantaneous
velocity of pattern displacements within the plane of image
formation. There are many methods to find out optical flow
out of which Gradient method is the basic method. But due to
the aperture problem gradient based method cant give the

exact result. To get the exact result differential based optical

flow algorithms (Horn-Schunck algorithm and Lucas-Kanade
algorithm) are used. In our approach we use Horn-Schunck
algorithm for segmenting the foreground object.
The Horn-Schunck algorithm assumes smoothness in the flow
over the whole image. Thus, it tries to minimize distortions in
flow and prefers solutions which show more smoothness.
The flow is formulated as a global energy functional which is
then sought to be minimized. This function is given for twodimensional image streams as:
(4)

the neighbors have been updated. The following iterative

scheme is derived:
(12)
(13)
where the superscript k+1 denotes the next iteration, which is
to be calculated and k is the last calculated result.

Where Ix, Iy and It are the derivatives of the image intensity

values

along

the

x,

and

time

respectively,

dimensions
is the optical

flow vector, and the parameter is a regularization constant.

This

functional

can

be

minimized

by

solving

the

associated multi-dimensional Euler-Lagrange equations. These

are
(5)
(6)
where

is the integrand of the energy expression, giving

(7)
(8)

denotes the Laplace operator. In

practice the Laplacian is approximated numerically using
finite

differences,

and

may

written
where

is a weighted average of

be
(9)

calculated in a

neighbourhood around the pixel at location (x,y). Using this

notation the above equation system may be written

and

F is said to be a Markov random field on with respect to a

neighbourhood system
if and only if the following two
conditions are satisfied:
P(f) >0 , for all f belongs to F , and

P(fi | fS -{i}) = P(fi| fNi), where S-{i} is the set difference, fS -{i}
(10) denotes the set of labels at the sites in S-{i} and
fNi = {fi| I belongs to Ni} stands for the set of labels at the
(11)

which is linear in

C. Markov Random Field

Markov random field models have become useful in
several areas of image processing. The success of Markov
random fields (MRFs) can be attributed to the fact that they
give rise to good, flexible, stochastic image models. The goal
of image modeling is to find an adequate representation of the
intensity distribution of a given image. What is adequate often
depends on the task at hand and MRF image models have been
versatile enough to be applied in the areas of image and
texture
synthesis,
image
compression,
restoration,
tomographic image and texture segmentation, texture
classification etc. A Markov Random Field (MRF) is a
graphical model of a joint probability distribution.Let F = {F1,
.,Fm} be a family of random variables defined on the set ,
in which each random variable Fi takes a value fi in . The
family F is called a random field. We use the notation Fi =
fi to denote the event that Fi takes the value fi and the
notation (F1=f1 ,.. ,Fm=fm) denote the joint event. For
simplicity, a joint event is abbreviated as F=f where f ={f1,
fm} is a configuration of F, corresponding to a realization
of the field. For a discrete label set , the probability that
random variable Fi takes the value fi is denoted , P(Fi = fi )
abbreviated P(fi) unless there is a need to elaborate the
expressions, and the joint probability is denoted
p(F=f)=P(F1=f1,,Fm=fm) and
abbreviated
P(f). For
a
continuous ,
we
have
probability
density
functions (p.d.f.'s), p(Fi = fi ) and p(F=f).

and may be solved for each pixel in

the image. However, since the solution depends on the

neighboring values of the flow field, it must be repeated once

sites neighboring i.
D. Proposed Algorithm
Moving object detection in video scenes is often a very
challenging problem in video surveillance application. The
method we proposed here can segment the moving object
very efficiently as well as tackled slow or fast moving

objects, indoor and outdoor lighting ,noise etc. To

demonstrate the proposed method for moving object
segmentation first we use GMM and optical flow
algorithm separately to extract the moving objects. GMM
segments the objects by establishing Gaussian distributions
for each pixel to adapt to background changes. Optical
flow optimize a function based on residuals from the
brightness constancy constraint, and a particular
regularization term expressing the expected smoothness of
the flow field and based on this flow field between frame i
and i+1 it segments the moving objects. Next, we will
merge the segmented objects obtained using GMM and
Optical flow. After this process Markov Random field
algorithm is applied for refining the segmented objects.

like ball . In outdoor sequence, first dataset is about a

moving car and second dataset is Walking: persons walking
in a corridor.
A. Indoor sequence
I.
First dataset

Fig. 2 (a) Original frames ; (b) Segmented moving objects

(a)

II.

(b)

Second dataset

(c)

(d)

Fig. 1 (a) Original Frame; (b) Segmented object using

GMM and optical flow; (c) Motion field; (d) Final
segmentation using MRF
Fig. 3 (a) Original frames; (b)Segmented moving objects
IV.

Experimental Results

This section presents the experimental results obtained

from two set datasets each of outdoor and indoor sequence
to evaluate the performance of the proposed method. The
test images in the experiment are 240 x 320 pixels wide .
The implementation of the proposed method runs at a speed
for 240x320 images on a personal computer with an AMD
32 bit 2.2 Ghz processor. We tested the proposed method
with indoor illumination changes, noise and sudden
movement of objects. First dataset of indoor sequence is
Intelligent room : Moving person from one side to another .
Second dataset of indoor sequence is a moving pendulum

B. Outdoor sequence
I.
First dataset

V.
b

Fig. 4 (a)Original frames; (b)Segmented moving objects

II.

Second dataset

Conclusion

This paper proposed an improved foreground moving object

segmentation method by incorporating both the features of
GMM and Optical Flow and finally refined the result by
Markov Random Field to obtain a accurate segmentation
result. The proposed algorithm is tested under slow and
sudden movement of objects, noise and indoor and outdoor
lighting condition. Experimental results show that our
proposed method can efficiently segment a variety of moving
objects efficiently. In some of the results the moving objects
part is over-segmented or under-segmented due to the shadow
and illumination problem. As extension of this work would be
the removal of shadow and illumination to obtain a better
segmentation result.

Reference
[1]

[2]

[3]

[4]

[5]

Fig. 5 (a) Original frames; (b)Segmented moving objects

[6]

[7]

C. Stauffer and W. Grimson, Learning patterns of activity using realtime tracking, IEEE Trans. Pattern Analysis and Motion Intelligence,
vol. 22, pp. 747-757, 2000 .
B.K.P Horn and B.G.Schunck, Determining optical flow, Artif. Intell.,
vol. 17, pp. 185-203, 1981
S.C.Pei and L.G. Liou, Motion based grouping of optical flow fields:
the extrapolation and subtraction technique , IEEE Trans. Image
Processing, vol. 6, pp.1358-1363,1997
J. Hirai, T. Yamaguchi and H. Hirada, Extraction of moving object
based on fast optical flow estimation , ICCAS-SICE, pp. 26912695,2009
D. Zhou and H. Zhang, Modeified GMM background modeling and
optical flow for detection of moving objects, International Conf. on
Systems, man and Cybernatics , vol. 3, pp. 2224-2229, 2005
W. Huang, I.Y.H. Gu and Q. Tian , Foreground object detection from
videos containing complex background, ACM MM ,2003
S.A. Berrabah , G.De Cubber, V. Enescu and H sahil , MRF-based
foreground detection in image sequences from a moving camera, ICIP,
pp. 1125-1128, 2006