Computers and Electronics in Agriculture 75 (2011) 147158

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier.com/locate/compag

Original paper

Support vector machine approach to real-time inspection of

biscuits on moving conveyor belt
S. Nashat a , A. Abdullah b , S. Aramvith c , M.Z. Abdullah a,
a

School of Electrical and Electronic Engineering, Engineering Campus, Universiti Sains Malaysia, 14300 Penang, Malaysia
Faculty of Information Sciences and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi Selangor, Malaysia
c
Department of Electrical Engineering, Faculty of Engineering, Chulalongkorn University, 10330 Bangkok, Thailand
b

a r t i c l e

i n f o

Article history:
Received 10 April 2010
Received in revised form 8 September 2010
Accepted 17 October 2010
Keywords:
Biscuit
Discriminant analysis
Machine vision
Multi-core processor
Image segmentation
Support vector machine

a b s t r a c t
An intelligent system for colour inspection of biscuit products is proposed. In this system, the state-ofthe-art classication techniques based on Support Vector Machines (SVM) and Wilks analysis were
used to classify biscuits into one of four distinct groups: under-baked, moderately baked, over-baked,
and substantially over-baked. The accuracy of the system was compared with standard discriminant
analysis using both direct and multi-step classications. It was discovered that the radial basis SVM
after Wilks was more precise in classication compared to other classiers. Real-time implementation
was achieved by means of multi-core processor with advanced multiple-buffering and multithreading
algorithms. The system resulted in correct classication rate of more than 96% for stationary and moving biscuits at 9 m/min. It was discovered that touching and non-touching biscuits did not signicantly
interfere with accurate assessment of baking. However, image processing of touching biscuits was considerably slower compared to non-touching biscuits, averaging at 36.3 ms and 9.0 ms, respectively. The
decrease in speed was due to the complexity of the watershed-based algorithm used to segment touching
biscuits. This image computing platform can potentially support the requirements of the high-volume
biscuit production.
2010 Elsevier B.V. All rights reserved.

1. Introduction
The bakery sector can be considered to be one of the most important sectors in food industry. Real-time inspection is very desirable
in this industrial sector since food products like biscuits are being
produced by millions each day. Like other manufacturing processes,
quality evaluation and sorting are two essential operations performed routinely in biscuit production. Among the many tests that
need to be carried out on biscuits is the measurement of colour
as colour indicates quality and defect. Colour is also an important
guide as biscuits will appear more appetising when its appearance
is optimised. Consumers expect to nd a constant colour for a same
brand of biscuit. Obtaining identical biscuits are difcult even
during a short baking period. This is due to the complexity of the
baking process where biochemical reactions and physical transformations give rise to biscuits with different shades of colours
(Moore, 1991). Therefore biscuit colour inspection occupies a major
role in the biscuit manufacturing, from raw dough to nished products. Commonly, in this industry, quality evaluation is being carried
out by human vision of some trained inspectors who make subjec-

Corresponding author. Tel.: +60 4 5996001; fax: +60 4 5941023.

E-mail address: mza@usm.my (M.Z. Abdullah).
0168-1699/$ see front matter 2010 Elsevier B.V. All rights reserved.
doi:10.1016/j.compag.2010.10.010

tive quality judgement as to the product status. This decision is

highly variable, and the process is tedious, labour-intensive, and
subjective.
In recent years, automated machine inspection systems have
been implemented in bakery for colour inspection. The use of robots
in food processing has many advantages over their human counterpart in terms of economy and performance (Abdullah, 2008).
Automated biscuit inspection would help to standardise the quality assessment of biscuit during the baking process. Although the
development of automatic visual inspection for bakery and agricultural products is steadily progressing in the last decade, most
machine vision systems are facing the restriction of real-time constraints and high computing volume for high-speed production.
Image processing of most high-speed commercial inspectors is generally based on thresholding and pixel counting, so that they are
effective at distinguishing products with large contrast or colour
differences. In the past there have been many developments on
the use of a digital imaging technique to inspect a single, isolated
and stationary agricultural products, such as the inspection of potatoes and apples (Tao et al., 1995), oil palm fruits (Abdullah et al.,
2002), rice and grains (Carter et al., 2006), citrus fruits (Blasco et al.,
2007), corns (Chen et al., 2009), and beef fat (Chen et al., 2010).
However, few of these developments are able to be implemented
for high-speed sorting of moving targets at an economical feasi-

148

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Fig. 1. Example of (a) non-touching and (b) touching biscuits. In both cases, top left is under-baked, top right is moderately baked, bottom left is over-baked and bottom
right is substantially over-baked.

ble cost. Image processing of a dynamically moving target requires

machine vision with high-data throughput and high-performance
microprocessors. This problem is further compounded due to the
fact that most practical applications require time consuming vision
tasks like image transformation and extraction. Fortunately, most
of these tasks have regular, repetitive computations with predictable data dependencies, making them well suited for parallel
paradigm. In order to exploit data-level parallelism in object recognition, multi-core processors are more and more used in recent
years to achieve real-time object recognition (Kim et al., 2010). For
instance Zhang et al. (2008) revealed that real-time operation over
30 frame/s for the scale invariant feature transform can be achieved
by parallelising on a multi-core system; Ach et al. (2008) developed
a multi-core processors for real-time detection of trafc signs in
different images by means of the Viola-Jones algorithm; Kyo et al.
(2008) present massively parallel single instruction multiple data
(SIMD) in the image pre-processing stage of object recognition; and
Zhao et al. (2009) exploited the thread-level and data-level parallelism of image segmentation technique based on Otsus algorithm.
Clearly from this review, multi-core technology is well suited to
solve some pattern recognition problems and to processing timevarying images. In this paper, we will present the in-depth analysis
of the multi-core implementation of the SVM-based articial classier for real-time inspection of stationary as well as moving objects.
Both isolated and partially occluded biscuits are investigated. The
proposed methods and procedures are applied to inspect biscuit
samples on a conveyor belt with adjustable speed.
2. Materials and methods
2.1. Biscuit colour grading
Unlike industrial and manufacturing sectors, to the best of
authors knowledge, no single standard for colour grading of biscuits exists. In this industry, the colour standard of biscuits is
determined by the individual company. Colour quality control in
most biscuit producers involved human inspectors while some rely
on a more objective measurement by using a colorimeter. However, this equipment is not suitable when the sample has colour
variations. Moreover, there are many reasons for considering the
problem of touching or partial recognition in biscuit processing. For
instance it is not always possible to keep biscuits from touching or
overlapping on a moving conveyor belt. Example of colour variation in biscuits highlighting touched and untouched cases is shown
in Fig. 1. In both cases, the biscuits are categorised into four distinct
groups reecting four degrees of baking: under-baked, moderately

baked, over-baked and substantially over-baked. It can be seen in

Fig. 1 that the colour is not uniformly distributed. This is due to
the temperature variation inside the oven, causing colours of biscuits to appear darker in some regions. Therefore, the challenge for
image processing software is to use this information as the basis
for colour inspection.
2.2. Machine vision system
The hardware used in this study consisted of a HP Workstation XW4600 with quad core CPU 2.5 GHz processor, 2 GB of RAM
and equipped with a colour frame grabber, an illumination system, cable, a charge-coupled device camera, and conveyor. The
frame grabber is of the 32-bit Meteor II type manufactured by
Matrox Electronic System Limited, Canada. This device is mounted
into the PCI slot of the workstation. A high quality 3-CCD Sony
XC-003P camera was used as the image capturing device with a
sustainable speed of 25 frames per second, captured at a spatial
resolution of 640 480 pixels. The camera comes with standard C-mount type optical lens connected to a frame grabber
via 2-m external BNC cable. The camera was mounted to a test
station, which comprised of iron holder xed at the height of
45.5 cm and at an angle of 90 mounted from horizontal arm. The
test station was illuminated using a white, Stocker Yale (USA)
ultra-high-frequency uorescent ring light, model 13 plus high
frequency steady light with a maximum oscillating frequency of
85 kHz. The light bulb was tted with a continuous light intensity control which allowed 10100% intensity adjustment. Standard
conveyor belt with adjustable speed was used in this study to simulate moving object detection. The schematic diagram of biscuit
inspection system showing all essential elements is depicted in
Fig. 2.
The proposed colour inspection system is divided into two main
steps: pre-processing and post-processing. In summary, the preprocessing step includes calibration of machine vision and the
image processing part. In this work the machine vision was calibrated using four colour samples manufactured by Labsphere Inc.,
USA. The samples are SCS-RD-010, SCS-GN-010, SCS-BL-010, and
SCS-YW-010, corresponding to red, green, blue, and yellow colour
standards respectively. The image processing part involves image
acquisition and smoothing, RGB to HLS colour transformation, and
image segmentation. The post-processing step includes dimensionality reduction of the segmented objects by using Wilks analysis,
and applying support vector machine to achieve classication.
In this study the image processing was performed using the
Matrox Image Library (MIL) version MIL 9.0 with visual C++ pro-

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

149

BNC Cable
Iron holder

1. Under-baked
2. Moderately baked
3. Over-baked
4. Substantially over-baked

3-CCD
Camera
Illumination
system

Biscuit
samples

Industrial HP
Workstation

Industrial conveyor

Frame
grabber

Fig. 2. Schematic diagram of biscuit machine vision inspection system.

gramming language incorporated. The parallel-processing is based

on Matrox multiple-buffering and multithreading techniques.
2.3. Colour space transformation
Colour is one of the most important features for biscuits assessment used in this recognition system. The images of biscuits were
taken by the CCD camera and represented in the three-dimensional
RGB colour space. Unfortunately, the RGB colour space used in computer graphics is device dependent, which is designed for specic
devices, e.g. cathode-ray tube (CRT) display. Hence, RGB colour has
no accurate denition for a human observer. Therefore, the effective way is to transform the RGB information sensed by the machine
vision to the Hue-Lightness-Saturation (HLS) colour space. The HLS
space was selected since it denes colour not only in the sense of
perceptual uniformity, but more signicantly, it matches to the way
that the human perceives colour (Camastra and Vinciarelli, 2008).
In this space, only the hue component h was analysed, since this
attribute directly characterises the colour properties of an object
and thus the degree of doneness of the biscuit. Hence, colour conversion process only involved transforming the RGB information to
a single hue buffer. Indirectly, this helped to compress information
for easier colour discrimination and manageable solution. For the 8bit machine vision, such a transformation is given mathematically
by Gonzalez and Wintz (1987):

h=

h=

360 cos
255
360

or


cos



(R G)2 (R B)(G B)

If B G


1

0.5[(R G) + (R B)]

0.5[(R G) + (R B)]

(R G)2 (R B)(G B)

(1)



255
360

If B < G

(2)

Eqs. (1) and (2) yield normalised hue values in the interval [0, 255].
The equation of hue can also be expressed using trigonometry identity, but Eqs. (1) and (2) are easier to visualise and superior in terms
of hardware implementation.

of the source image was internally generated then the threshold

value was set to a minimum value between the two most statistically important peaks in the histogram, on the assumption that
these peaks represent the object and the background. If the histogram contains only two peaks, the threshold value was set to
the minimum value between these peaks. If the histogram contains
more than two peaks, then the threshold value was set between the
two most prominent peaks. In this way the biscuit images could be
separated from the background. The area of segmented image was
then calculated in order to distinguish between touching and nontouching biscuits. This could easily be done since the surface area of
each biscuit was relatively constant. It was heuristically discovered
that the surface area of the biscuit lies in the range between 39,870
and 39,990 pixel square. Therefore, it was possible to distinguish
touched from the untouched cases using direct thresholding. If the
area was smaller than the threshold value, then, the biscuits were
considered untouched. In this case, image processing only involved
smoothing and the removal of artefacts or noise. This was achieved
by means of morphological erosion technique. For touched cases,
image processing required four more additional steps. Sequentially
they are (i) edge detection, (ii) watershed transformation, (iii) morphological erosion, and (iv) distance transformation. For the biscuit
image that differs signicantly from the background, the Sobel
operator (Sobel, 1970) was applied to detect the edge of biscuit.
Then the watershed transformation was implemented to the gradient image (Vincent and Soille, 1991). The watershed transformation
considers the gradient magnitude of an image as a topographic surface. If the objects have well-dened edges, an edge detection will
produce a maximum along the edges of each object. These maxima
will dene each object as a catchment basin since they produce
a minimum in each object. A watershed transformation will then
label the catchment basins, effectively segmenting the image. After
that morphological erosion was applied to smooth the image and
remove artefacts. Then, watershed and the distance transformation
were implemented in order to separate touching objects and reduce
noise due to overlapping. The blob analysis was then used to compute the centre of gravity of each object. Finally, image cropping
was applied to the original image in order to obtain the region of
interest. The owchart in Fig. 3 summarises the image processing
sequence of touching and non-touching biscuits.

2.4. Image segmentation and object detection

2.5. Dimensionality reduction by Wilks  analysis
To separate an image of biscuit from the background, the
combination of auto thresholding and watershed transformation
techniques were proposed. The auto thresholding method locates
the two peaks in the histogram corresponding to the background
and object of the image (Otsu, 1979). In so doing, the histogram

Similar to other image-based articial classiers the resulting

hue vectors from the above image processing algorithm are still
too large to allow fast and accurate detection. Therefore, it is very
desirable to use fewer hue values, preferably those with highest dis-

150

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Support vectors

Start

Margin

Grab image
Auto thresholding

Class (+1)

Calculate area for

each object

Area > T
Non-touching No
objects

Morphological
erosion

Yes

Touching
objects

Hyperplane
Edge detection of
original image
Watershed
transformation
Morphological
erosion

Count number of
objects and calculate
centre of gravity

Class (-1)

Separate touching using

distance transformation

Image cropping
End
Fig. 3. Image processing for touching and non-touching biscuits. The threshold
value T was set to 40,000 pixel square.

criminant powers. Hence, a sequence of operations is needed before

performing classication. The most commonly used technique is
Wilks selection criteria, which is also known as the stepwise discriminant analysis (Rencher, 2002). In summary, this method works
by examining each hue value in the set containing all hue variables
hp , iteratively updating and deleting each variable until a subset
containing a best hm
s is produced from the full available of hs . The
decision to enter or to delete each variable is based on F-statistic
together with F-to-enter and F-to-remove values. The mathematical derivations and technical description of Wilks analysis are
described in our earlier publication (Abdullah et al., 2001).
2.6. Classication using SVM
Support vector machine is a supervised learning algorithm
developed by Vapnik and others (Cortes and Vapnik, 1995; Vapnik,
1998). It has been used extensively for a wide range of applications
with excellent empirical performance (Karimi et al., 2006; Adankon
and Cheriet, 2009). The fundamental idea of SVM is to construct a
hyperplane as the decision line, which separates the positive (+1)
classes from the negative (1) ones in the binary classication with
the largest margin. This is illustrated in Fig. 4.
The classication of biscuits into four quality levels can be considered as a collection of binary categorisation problems. Suppose
there are l samples of biscuit in the training data corresponding
,
to the two groups, and each sample is denoted by a vector h
i
(i = 1, . . ., l), which represents the selected colour features of the
biscuit. Therefore the training data can be represented by the set of

Fig. 4. SVM uses hyperplane margin to separate positive from negative classes.

 , y ), where h
 is the hue and y {+1, 1} indicates the
vectors (h
i
i
i
i
class label. The classication of biscuit samples can be considered
 y using
as the task of determining a classication function f : h
i
i
training data. Subsequently, the classication function f is used to
 ) > 0, the input vector h

classify the unseen test data set. If f (h
i
i
is assigned to the class yi = +1, otherwise class yi = 1. This technique can be extended to classication involving more than two
groups using SVM algorithm such as the Directed Acyclic Graph
(Platt et al., 2000). Essentially, this algorithm works by rstly collecting the data from each classier, and secondly, assembling these
data to form a graph or a tree comprising of several nodes with each
node representing result from each binary classier. Therefore, it is
theoretically plausible to perform more than 2-class classication
by expanding and adding more nodes to the tree. The details are
described in Section 2.7.
For the linearly separable training vectors, the classication
function f has the following form:
 + b)
 ) = sgn(w
 Th
f (h
i

(3)

 is the normal to the hyperplane, and b is a bias term which

where w
should satisfy the following conditions:
 + b) 1,
 Th
yi (w
i

i = 1, . . . , l

(4)

In operation, the SVM attempts to nd the optimal hyperplane

that maximises the separation margin between positive and neg thus the optimal separating
ative samples. The margin is 2/||w||,
 subject to con T w),
hyperplane is the one minimising ((1/2)w
straints (4), which is a convex quadratic programming problem.
For the non-linear case when the hyperplane can only separate
the groups algebraically but not strictly separate them, a differ are mapped
ent strategy is needed. In this case, the input vectors h
i
into a high dimensional feature space H by using suitable kernel
 ). Two popular kernels which are the polynomial
,h
function k(h
i
j
kernels and the Gaussian radial basis function (RBF) were used in
the study. Respectively, they are dened as:
 ,h
 ) = (h
h

k(h
i
j
i j + 1)

(5)

and
 ) = exp
 ,h
k(h
i
j

 h
 ||2
||h
i
j

2 2

where d is the degree of polynomial kernel.

(6)

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

The classication function then has the following form in terms

of kernels:

 = sgn
f (h)

1
2
3
4

 , h)
 +b
i yi k(h
i

(7)

151

1. Under-baked
2. Moderately baked
3. Over-baked
4. Substantially over-baked

1vs4
not 1 not 4

i=1

2
3
4

where k is a kernel function, b is a bias term, and i is the Lagrange

multiplier coefcient obtained by solving the quadratic programming problem. Based on KarushKuhnTucker (KKT) conditions
 with the corresponding
of quadratic programming, only those h
i
coefcients i > 0 in Eq. (7) are called support vectors (SVs), which
are used to make the decision in the classication problem. Platt
(1999) describes the techniques and procedures involved in KKT
programming. Interested readers are referred to this reference for
further details. Finding coefcients i is equivalent to maximise the
function as follows:
W () =

l

i=1

1
 ,h
 )
yi yj k(h
i
j
i j
2
l

not 2

3
4

1
2
3

2vs4
not 4

2
3

3vs4

1vs3
not 3

not 1

1
2

2vs3

1vs2

(8)

i=1 j=1

Fig. 5. The acyclic graph for 4-group classication via DAG strategy. Here 1 indicates under-baked, 2 moderately baked, 3 over-baked and 4 substantially
over-baked. The abbreviation vs stands for versus.

Under the constraints:

0 i C, (i = 1, . . . , l) and

yi i = 0

(9)

2.8. Experimental procedures

i=1

where C is a non-negative regularisation parameter.

For some data sets, the SVM may not be able to nd a separating
hyperplane in feature space. Therefore a regularisation parameter C was used to control the trade-off between maximising the
margin and minimising the training error. Entirely specifying a
support vector machine requires setting two parameters: the kernel function and the magnitude C for violating the soft margin.
The selections of these parameters are dependent on the specic
modelled data. In this study the sequential minimum optimisation
(SMO) algorithm was used to train the SVM algorithm and SVs are
analytically obtained by solving quadratic programming problem.
The details of SMO algorithm are described elsewhere (Platt, 1999).
2.7. Multi-class SVM with directed acyclic graph
For the problem of N-class SVM (N = 4), Platt et al. (2000) proposed a novel algorithm for multiclass classication based on
placing two-class classiers into nodes of a directed acyclic graph
(DAG). The graph used a rooted binary DAG, a root DAG that has
a unique node with no arrows pointing into it, and other nodes
which have either 0 or 2 arcs leaving them, to be a class of function
in classication tasks, as shown in Fig. 5.
This algorithm constructs N(N 1)/2 internal nodes, each one
labelled with an element of a Boolean function. The DAG can be
performed using a list, where each node eliminates one class from
the list. The implementation is initiated with a list of all classes. A
test point is evaluated against the decision node that corresponds
to the rst and last elements of the list. If the node prefers one of
the two classes, the other class is eliminated from the list. The DAG
continues with testing until only one class remaining in the list, at
this stage, the algorithm will terminate and the state of the list is
the entire state of the system. The classication of biscuit samples
into four groups was implemented by using DAG algorithm. Here,
we refer to the decision node distinguishing groups i and j as the
i j node. Such a DAG is obtained by training each i j node only
on the subset of training points labelled by i or j. The nal class
decision is derived by using the DAG architecture as shown in Fig. 5.
The details of the DAG SVM are described elsewhere (Nashat and
Abdullah, 2010).

In this colour inspection system 400 samples of biscuit were

inspected and categorised to either training or independent test
sets. The rst 200 samples were allocated to the training set, comprising 50 samples for each group. These samples were then imaged
and latter used to train the machine vision system for colour inspection and recognition. Another 200 samples were inspected by the
inspectors and categorised as the independently set, comprising 50
samples for each group. This set was used to independently assess
the accuracy of the machine vision system. Initially, the captured
training set was converted from RGB to HLS colour space using Eqs.
(1) and (2). After that, image segmentation algorithm described
above was implemented to crop 80% of biscuit sample producing a
region of interest. Then the resulted hue population ranging from 0
to 59 was used as input variables to the machine vision system. In
this stage Wilks algorithm was invoked with several threshold
options in order to extract the maximum number of potent variables. Once the optimum or potent hue subset has been located,
the system established the SVM classier by scaling each input
hue value and applying SMO algorithm. The colour inspection system rst mapped the scaled hue vector to high dimensional feature
space by using one of the kernel functions in Eqs. (5) and (6). Then
the SMO algorithm was implemented to train the system and established criteria from the image multiple samples by calculating all
variables useful in classication. Finally, the classication output
is then processed using DAG algorithm which uniquely classied
sample into one of the four groups.
After the training was completed, the machine vision system
was switched from training to test mode. In this mode, the system
was used to reclassify the training samples in order to assess its
accuracy, repeatability and consistency. After this, the system was
used to classify all samples belonging to the independent test set.
In this way, the variability in grading between inspectors could be
compared, and the accuracy between machine vision system and
inspectors could also be investigated. All biscuit samples were manually classied and sorted by hand using semi-trained inspectors
following the same methods and procedures discussed previously.
In this study, results obtained from DAG algorithm were compared with DA based classier using both full and reduced features.
After that, the accuracy of the system was assessed on moving test
samples using different speeds in real-time.

152

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

2.9. Parallel image processing

To achieve real-time object recognition with considerably lower
power, we considered the multi-core architecture comprising of
4 processing units with 6 MB of level 2 cache memory. The basic
idea is to partition the image into specic parts and simultaneously processing them on different processors. Respectively, the
multiple-buffering technique and the multithreading strategy are
implemented to manage the image capturing cycle and synchronise
the execution of the program in parallel threads. These methods are
described in the following sub-sections briey.
2.9.1. Multiple-buffering technique (MBT)
The key component in reaching real-time implementation is the
Matrox dynamic simulator featuring the multiple-buffering strategy. This technique involves grabbing images and storing them into
buffers while processing previously grabbed images. In this way
MBT optimises wasted CPU cycles and reduces the memory cost
due to managing inconsistent frame complexity. This technique
allows images to be grabbed and processed concurrently. It also
permits for robustness when there is variability in processing time
from image to image (Hornberg, 2006).
In this study, multiple buffering is performed by rst allocating
list of buffers used to hold series of sequentially grabbed images,
and to allow for the processing of the buffers concurrently as they
are being grabbed. This grabbing is initiated by a start signal and
the process continued until the stop signal is activated. In this case,
grabbing is implemented asynchronously which allows other processing functions to be executed while images are being acquired.
The technique goes round-robin through the list of buffers in a continuous acquisition. However, care must be taken to ensure that the
average time to process a frame is not greater than the frame rate of
the camera. Therefore, the design in Fig. 6 is adopted showing the
sequence of image grabbing and storing. To avoid memory access
conicts between the video interface and the worker threads, two
buffers of at least one for grabbing and one for processing are
required. In this experiment, n (2 n 22) buffers are used to hold
the images and the processing function is called n times to process
them. In this way complex image processing tasks can be realised
as speed equals to the frame rate of the acquisition hardware.
2.9.2. Multithreading technique
Multithreading technique and multi-core processing are two
powerful strategies to take advantage of parallelism in the applications in order to improve the systems performance. Computer
vision applications have the potentials to run much faster by
utilising more powerful multi-core systems. On the other hand,
algorithms and applications must be tuned to allow multi-core
processor to exploit their inherent parallelism. Multithreading
technique has the ability to perform multiple operations simulta-

Grab 2

Grab 1

Processing 1

Task 1
Thread 1

Task 2

Task 3

Thread 2

Thread 3

Frame
grabber

Buffer 1
Processing 1

Camera

Buffer n-1
Processing n-1
Buffer n
Processing n

Fig. 6. Block diagram of the multiple-buffering strategy for real-time grabbing and
image processing.

neously in the same process. This can be done by creating different

threads to ensure sequential execution of operations within the
same thread while allowing simultaneous yet independent execution of operations in other threads. Here, the multithreaded
programming is implemented using MIL tools for multiprocessing.
In this work, multithreading was implemented by assuming that
there are maximum of four samples in the eld view. It is possible
to have more than four samples but the image resolution would
be signicantly reduced and the classication would be less accurate. The algorithm works by rstly extracting important features
from each sample in the image, and secondly, passing this information simultaneously to all distinct SVM classication threads in
parallel. After the classication has completed, each thread gave
a result corresponding to each sample in the image. Fig. 7 shows
the structure of queuing tasks of four-threads SVM used in this
study.
Prior to multithreading, the image is segmented into few subimages corresponding to the number of biscuit sample in the eld
of view. Resultantly, the background is also ltered leaving the
desired features only. The multi-core processor used the feature
space to count the number of sample, and resultantly, created
the number of compute threads in the node. One of the compute
threads is assigned as a master, which allocates work to all other
compute threads. Each thread independently performs the image

Grab n

Processing 2

Task 4
Thread 4

Processing n

Task 1
Thread 1

Task 2

Task 3

Thread 2

Thread 3

Fig. 7. Structure of queuing tasks of the inspection system and with four SVMs threads.

Task 4
Thread 4

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

153

the multi-threaded data ow structure implemented in this

study.

Start

3. Results and discussion

Grab image
3.1. Colour inspection

Store image into buffer

Image segmentation and object

detection

If no. of
object 1

No

Yes
Master thread crops the
grabbed image into sub-images
Master thread forks several
threads one for each object
Each thread picks a task and
processes the sub-image
Each thread writes result into
memory

Each thread joins Master thread

Master thread
displays the results

Free buffer

No

Stop
grading
Yes
End

Fig. 8. Flowchart of a real-time colour inspection system based on multithreading

technique for parallel processing.

post-processing like dimensionality reduction and object recognition. Results are assembled by the master thread and displayed on
the monitor screen or written into a text le if necessary. After
completion of the image inspection task, the master thread frees
all buffers and the threads. Then, it enters a stand-by mode and
ready to process next image. The owchart in Fig. 8 summarises

The edge images corresponding to non-touching and touching

biscuits in Fig. 1 are shown in Fig. 9. The formation of four disjointed regions corresponding uniquely to four biscuits in the scene
is clearly visible in this gure. A cross-hair indicates the centre of
each biscuit in the image. This result indicates that auto thresholding performs well for non-touching biscuits while the watershed
transformation is effective in dealing with touching biscuits. Following segmentation, the hue values of each biscuit are extracted
and then plotted as a bar graph or histogram. Example of the hue
histogram is shown in Fig. 10. This gure is plotted by averaging
over 50 samples of biscuits for each group category. The histogram
shows a typical hue distribution of biscuits which contains many
local maxima and minima. It was also heuristically discovered that
the frequency distribution of hue depended strongly on the background colour, producing an image with optimal contrast when
background hues ranged from green to blue. Hence, the cyan background corresponding to the hue value 180 was used to capture
the biscuits images.
Careful examination of Fig. 10 reveals some unique characteristics of biscuits. First, they are not truly brown, based on quantitative
hue values, but rather show colours that tend toward yellow and
light or dark brown. Clearly, the colour of the under-baked group
is skewed towards the yellow and light brown region, peaking at
41; whereas the colours of the over-baked and substantially overbaked groups are skewed towards the dark-brown region. For these
two groups, the hue reached maximum values at 21 and 16, respectively. The hue distribution of moderately baked group lies between
light and dark brown region, peaking at approximately 20. Secondly it can be seen that the hue distributions for different groups
of biscuits overlap each other; no single threshold exists that can
distinctively separate one group from another. Thirdly, the hue
values for the four types of biscuits are approximately normally
distributed, ranging from 16 to 43 for the under-baked group, from
14 to 43 for the moderately baked group, from 12 to 30 for the overbaked group, and from 4 to 23 for the substantially over-baked
group. This is in agreement with expected values, given that hue
values that range from 0 to 60 fall between the red and yellow
regions.
However, not all these values are necessarily independent or
uncorrelated variables. In fact, some of them are redundant and do
not have signicant discriminating power. Therefore, these variables need to be eliminated, since such an elimination will not only
reduce the dimensionality of independence of the variables among
themselves, but also, and more importantly, it produces a best subset of variables leading to an optimal rate of correct classication
(Smith and Nakai, 1990; Abdullah et al., 2004).
3.2. Wilks  analysis
The important element in this colour inspection system is to
locate a hue subset that contains signicant input variables. Referring to Fig. 10, the hue values of four different groups of biscuit
samples fall in the range from 0 to 59. Hence, altogether there are 60
variables, which are statistically independent and useful for analysis. These variables were submitted to Wilks analysis in order to
detect both the insignicant and highly signicant hue variables. In
the analysis, the F-to-remove and the F-to-enter variables were set
to default values of 2.71 and 3.84, respectively. The algorithm iteratively took eight steps to converge, producing a subset containing

154

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Fig. 9. Segmented biscuits and their centres of gravity corresponding to image in Fig. 1: (a) non-touching biscuits and (b) touching biscuits. Here a(i) and b(i) are edge images
after auto thresholding; b(ii) is the edge image after watershed-based segmentation; a(ii) and b(iii) are the centres of gravity of a(i) and b(ii), respectively.

8 principal hues. This corresponds to a loss of 86.67% in variation.

The 8 principal components produced here were:
hs = (19, 20, 23, 29, 39, 40, 42, 43)

(10)

Using these potent variables, the machine vision system was

trained using biscuit samples in the training set. The classication
efciency of the system using 8 principal hues was examined by
studying the Mahalanobis distances, shown canonically in Fig. 11.

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

155

Fig. 10. Hue distributions of four different groups of biscuits averaged over 50 samples for each group category.

Table 1
Real time classication results of independent test samples corresponding to different conveyor speeds using SVM-R classier with Wilks method.
Conveyor speeds

Classication results
Under-baked

10 m/min
9 m/min
8 m/min

Moderately baked

94%
96%
96%

98%
98%
98%

Over-baked

Substantially over-baked

90%
88%
90%

Average accuracy

100%
100%
100%

95.5%
95.5%
96%

Table 2
Total processing time of non-touching biscuits comparing sequential and parallel processing for different classiers.
Classiers Sequential processing

Parallel processing
Wilks method

Direct method

Wilks method

Direct method

No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms)
SVM-R
SVM-P
DA

4
4
4

29.44
29.51
24.14

4
4
4

28.26
28.19
24.10

Functions 1 and 2 in this gure refer to the canonical discriminant functions derived from the canonical correlation analysis
which were used to examine the relationship between the variables
(Morison, 2005).
This graph clearly demonstrates the formation of four major
groups. It shows the existence of the hyperplane which strongly

4
4
4

9.68
9.70
8.87

4
4
4

9.34
9.30
8.25

separates under-baked and substantially over-baked groups. In

contrast, the hyperplane separation is weak for over-baked and
moderately baked groups. It can also be seen that these groups
are disjointed and convex, and the hyperplane separation is algebraically non-linear. It was also observed that the non-linearity
increased when independent test samples were used.

Table 3
Total processing time of touching biscuits comparing sequential and parallel processing for different classiers.
Classiers Sequential processing

Parallel processing
Wilks method

Direct method

Wilks method

Direct method

No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms)
SVM-R
SVM-P
DA

4
4
4

62.44
62.48
58.28

4
4
4

61.35
61.30
58.22

4
4
4

36.70
36.86
36.20

4
4
4

36.50
36.43
35.93

156

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Fig. 11. Discrimination separations by four biscuit groups produced by using Wilks
method. Data are averaged from 50 samples of each group.

Fig. 12. Classication results of training samples comparing SVM and DA classiers
by using Wilks and direct methods.

3.3. Classiers analysis

The experimental results on colour recognition of the SVM and
DA algorithms are summarised in Figs. 12 and 13, corresponding
to the results from classication of the training and independent
test samples, respectively. In these gures, results from the direct
application of SVM and DA are also presented, serving as reference
to Wilks analysis. In direct method, all hue variables ranging from

Fig. 13. Classication results of independent test samples comparing SVM and DA
classiers by using Wilks and direct methods.

0 to 59 are used for classication, whereas 8 principal hue values

are selected with Wilks method. In this experiment, polynomial
and RBF SVM classiers are trained and tested using the kernels
dened in Eqs. (5) and (6).
A range of parameters for the polynomial and RBF SVM classiers are selected and tested to eliminate any biased performance of
the SVM that may be caused by inappropriate choice of parameters.
In this experiment, the best parameter of polynomial SVM
(SVM-P) is selected to be degree d = 3 for direct and Wilks methods in all trials. In this work, the sigma () and penalty (C) variables
were chosen experimentally through a search algorithm. In so
doing, the seed values of 0.001 and 1 respectively was used for C and
. These seed points were heuristically decided. During searching,
the algorithm solved Eq. (7) repeatedly and the C and  values that
resulted in the best accuracy were selected. In this case the sigma
values selected for RBF SVM (SVM-R) were 58 and 2 for direct and
Wilks methods, respectively. Meanwhile the best penalty variable C was 1000. This penalty parameter improved the results of the
system and solved non-separable data set by using a soft margin.
Results are shown in Figs. 12 and 13.
The results using training samples produced high performance
accuracy which is clearly seen in Fig. 12. Here SVM-R and SVM-P
classiers yielded best classication rate of 100% with direct and
Wilks methods, whereas DA classiers yielded 99.5% and 99% by
using the same methods, respectively. Results using test samples
produced the same trend. It can be seen from Fig. 13, that the SVMR classier with Wilks method, resulted in the best classication
rate of 96.5%, compared 95.5% SVM-P and 94% DA. The results of
direct method in Fig. 13, presented a best performance for SVM-R
with correct rate 95%, comparing with the SVM-P and DA classiers,
which yielded 92% and 93%, respectively. In general the classication rates for SVM-R are consistently higher compared to SVM-P or
DA.
These results also indicated that discarding of variables, which
are correlated highly with those retained, guarantees that the
deleted variables are, in fact, redundant in the variable space.
Hence, the classiers after Wilks analysis is more precise in
classication compared with direct application of SVM and DA algorithms.
Results from these experiments also demonstrated the difculties in machine vision distinguishing over-baked from moderately
baked samples. Referring to Fig. 13, the average success rate for
the over-baked samples with Wilks method is 90% compared to
96.5% for other classes. These results are consistent with the canonical plot in Fig. 11 which showed that the hyperplanes separating
over-baked samples to other samples particularly the moderately
baked ones are relatively small. Considering this and the fact that
misclassication of the vision system varied from 0 to 6% and average on the order of 3% for individual class, the above results may
therefore suggest that machine vision classication may be equal or
better than semi-trained judgement in accuracy and consistency.
Comparing Figs. 12 and 13, it can also be seen that the good
classication results of the training samples are not reproduced in
classication of the test samples. Therefore the 100% or 99% average
success rate of training samples is articial, since the rate of correct classication is based against the best subset in this system.
In other words, the machine vision system seemed to memorise
the colour patterns in the training set instead of generalising them.
This observation is also in agreement with results on classication
and quantication of adulterants in marple syrup by multivariate
analysis of Fourier transforms infrared and near-infrared proles
(Paradkar et al., 2002).
Based on the stationary classication results, SVM-R classier
with Wilks analysis was chosen to inspect moving biscuits placed
on the conveyor belt. The experiment was conducted on biscuits
moving at three different speeds: (i) 8 m/min, (ii) 9 m/min and

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

(iii) 10 m/min. In this inspection, the machine vision system was

trained and tested using the same samples and illumination setup as those used in the stationary classication. Here, the biscuit
images were acquired continuously from the centre of the eld of
view by detecting the centre of each moving sample.
Table 1, shows the results of the dynamic classication using
independent test samples corresponding to the different conveyor
speeds. Expectedly, it can be seen from this table that there is a
slight decrease in the accuracy compared to stationary inspection.
The average correct classication attained by the system for moving samples is consistently higher than 95.5%. There is no signicant
change in the accuracy when the speed of the conveyor is varied
from 8 m/min to maximum adjustable speed of 10 m/min. Furthermore, the same correct classication is observed when the system
is used to inspect touching biscuits. In addition to accuracy, the
total processing times were also considered. These are tabulated in
Tables 2 and 3 for non-touching and touching biscuits, respectively.
It can be seen from these tables that the multi-threaded program
is more efcient when processing non-touching biscuits, resulting
in the speedup of more than 3.0. The average speedup of touching
biscuits is less than 1.7. The maximum speedup attainable by this
system is approximately 4. The slightly less than the ideal speedup
is attributed to the fact that the image segmentation procedure is
still performed in sequential. On average the utilisation of CPU cores
is measured as 65%. This result indicates multithreading improves
parallelism and the multiple processing cores are highly utilised.
Results comparing single and multi-core processing are also summarised in Tables 2 and 3. In both cases, the DA is slightly faster than
SVM even though SVM is more superior in terms of accuracy and
performance. On average the accuracy of SVM-R stands at 96.5%
compared to 94% for DA. Therefore, the slight reduction in speed
of the former is compromised with its increase in accuracy. It can
also be seen from these tables that the image processing of touching biscuits is at least 4 times longer than non-touching biscuits. In
addition to the auto thresholding, image processing of touching biscuits involved the watershed-based segmentation technique which
is wellknown for its extreme computational demand (Vincent and
Soille, 1991). Hence, the image processing for touching biscuits is
more intensive compared to non-touching biscuits. Based on SVMR classication results running on four samples, the vision system
should have a theoretical throughput of 428 biscuits/s or 110 biscuits/s for non-touching and touching cases, respectively. These
results indicate that the total inspection process of several samples
can be realised as speed less than the frame rate of the acquisition
hardware. In summary the algorithm is able to perform inspection at 10 m/min with correct classication of more than 95.5%.
This image processing capacity has the potential to fully satisfy the
general requirements of a fast biscuit manufacturing.

4. Conclusion
Quality evaluation of bakery products has a major role in food
industry. In this study a colour-based inspection system has been
proposed and shown to have a good potential to classify biscuit
samples using SVM and DA classiers. Results from this study indicate that SVM generally performs better than DA for both direct
and Wilks classications. It was discovered that the SVM-R
after Wilks resulted in correct classication of 96.5% which is
the highest compared to other classiers. Parallel processing was
implemented based on Matroxs multiple-buffering technique and
multithreading programming. These features combined provide
an efcient communication scheme, resulting in the increased in
image processing speedups by a factor of 3. There was no noticeable change in the accuracy when the system was used to inspect
moving biscuits. Like the stationary case, the highest classication

157

was recorded by SVM-R at a speed of 10 m/min. This articial classier has a potential for use in routine inspection of biscuits and other
bakery products. This articial classier has a potential to increase
inspection speeds beyond human operators when implemented on
a multi-core processor. It is suitable for use in high volume production like biscuit processing considered in this example.
Acknowledgements
This work has been supported by the Universiti Sains Malaysia
Research University Grant 814012. The authors also would like to
acknowledge partial support of this research through the AUN-Seed
Net Program 449/USM/2009.
References
Abdullah, M.Z., Guan, L.C., Mohd-Azemi, B.M.N., 2001. Stepwise discriminant analysis for colour grading of oil palm using machine vision system. Transactions of
IChemE 79 (C), 223231.
Abdullah, M.Z., Guan, L.C., Mohamed, A.M.D., Noor, M.A.M., 2002. Colour vision
system for ripeness inspection of oil palm elaeis guineensis. Journal of Food
Processing and Preservation 26 (3), 213235.
Abdullah, M.Z., Rahman, A.S.A., Shakaff, A.Y.M., Noor, A.M., 2004. Discrimination
and classication of Eurycoma longifolia Jack in medical food by means of a
DSP-based electronic taste sensor. Transactions of the Institute of Measurement
and Control 26 (1), 1939.
Abdullah, M.Z., 2008. Quality evaluation of bakery products. In: Sun, D.-W. (Ed.),
Computer Vision Technology for Food Quality Evaluation. Elsevier, London, pp.
481522.
Ach, R., Luth, N., Techmer, A., 2008. Real-time detection of trafc signs on a multicore processor. In: Proceedings of the IEEE Intelligent Vehicles Symposium,
Eindhoven, The Netherlands, pp. 307312.
Adankon, M.M., Cheriet, M., 2009. Model selection for the LS-SVM application to
handwriting recognition. Pattern Recognition 42 (12), 32643270.
Blasco, J., Aleixos, N., Molt?, E., 2007. Computer vision detection of peel defects in
citrus by means of a region oriented segmentation algorithm. Journal of Food
Engineering 81 (3), 535543.
Camastra, F., Vinciarelli, A. (Eds.), 2008. Machine Learning for Audio Image and Video
Analysis Theory and Applications. SpringerVerlag, London.
Carter, R.M., Yan, Y., Tomlins, K., 2006. Digital imaging based classication and
authentication of granular food products. Measurement Science and Technology
17 (2), 235240.
Chen, X., Xun, Y., Li, W., Zhang, J., 2009. Combining discriminant analysis and neural
networks for corn variety identication. Computers and Electronics in Agriculture 71 (1), S48S53.
Chen, K., Sun, X., Qin, Ch., Tang, X., 2010. Colour grading of beef fat by using computer
vision and support vector machine. Computers and Electronics in Agriculture 70
(1), 2732.
Cortes, C., Vapnik, V., 1995. Support-vector network. Machine Learning 20 (3),
273297.
Gonzalez, R.C., Wintz, P.A. (Eds.), 1987. Digital Image Processing. Addison-Wesley,
Reading, MA, USA.
Hornberg, A. (Ed.), 2006. Handbook of machine vision. Wiley-VCH Verlag GmbH &
Co. KGaA, Germany.
Karimi, Y., Prasher, S.O., Patel, R.M., Kim, S.H., 2006. Application of support vector
machine technology for weed and nitrogen stress detection in corn. Computers
and Electronics in Agriculture 51 (12), 99109.
Kim, J.-Y., Oh, S.-J., Lee, S.-J., Kim, M.-S., Oh, J.-W., Yoo, H.-J., 2010. An attention
controlled multi-core architecture for energy efcient object recognition. Signal
Processing: Image Communication 25 (5), 363376.
Kyo, S., Okazaki, S., Koga, T., Hidano, F., 2008. A 100 GOPS in-vehicle vision processor
for pre-crash safety systems based on ring connected 128 4-way VLIW processing elements. In: Proceedings of the IEEE Symposium on VLSI Circuits, Hawaii,
USA, pp. 2829.
Moore, C.A. (Ed.), 1991. Automation in the Food Industry. Glasgow, Blackie, New
York.
Morison, D.F. (Ed.), 2005. Multivariate Statistical Methods. Thomson Brooks, California.
Nashat, S., Abdullah, M.Z., 2010. Multi-class colour inspection of baked foods featuring support vector machine and Wilks analysis. Journal of Food Engineering
101 (4), 370380.
Otsu, N., 1979. A threshold selection method from gray-level histograms. IEEE Transactions on Systems Man and Cybernetics 9 (1), 6266.
Paradkar, M.M., Sivakesra, S., Irudayaraj, J., 2002. Discrimination and classication of
adulterants in marple syrups with the use of infrared spectroscopic techniques.
Journal of Science of Food and Agriculture 83 (7), 714721.
Platt, J.C., 1999. Fast training of support vector machines using sequential minimal optimisation. In: Schkopf, B., Burges, C.J.C., Smola, A.J. (Eds.), Advances in
Kernel Methods: Support Vector Learning. MIT Press, Cambridge, MA, pp. 185
208.

158

S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Platt, J.C., Christiani, N., Shawe-Taylor, J., 2000. Large margin DAGs for multiclass
classication. Advances in Neural Information Processing Systems, vol. 12. MIT
Press, Cambridge, pp. 547553.
Rencher, A.C. (Ed.), 2002. Methods of Multivariate Analysis. John Wiley & Sons,
Canada.
Smith, M.A., Nakai, S., 1990. Classication of cheese varieties by multivariate analysis
of HPLC proles. Canadian Institute of Food Science and Technology Journal 23
(1), 5358.
Sobel, I.E., 1970. Camera Models and Machine Perception, AIM-21. Stanford Articial
Intelligence Lab, Palo Alto, USA.
Tao, Y., Heinemann, P.H., Varghese, Z., Morrow, C.T., Sommer, H.J., 1995. Machine
vision for colour inspection of potatoes and apples. Transactions of the American
Society of Agricultural Engineers 38 (5), 15551561.

Vapnik, V. (Ed.), 1998. Statistical Learning Theory. John Wiley & Sons, New York.
Vincent, L., Soille, P., 1991. Watersheds in digital spaces an efcient algorithm based
on immersion simulations. IEEE Transactions on Pattern Analysis and Machine
Intelligence 13 (6), 583598.
Zhang, Q., Chen, Y., Zhang, Y., Xu, Y., 2008. SIFT implementation and optimisation
for multi-core systems. In: Proceedings of the IEEE International Symposium on
Parallel and Distributed Processing IPDPS, FL, USA, pp. 18.
Zhao, J., Yang, Y.M., Li, G., 2009. Real-time image processing system based on
multi-core processor. In: Proceedings of the IEEE 3rd International Symposium
on Intelligent Information Technology Application IITA, Nanchang, China, pp.
329332.