PATTERN RECOGNITION & ANALYSIS

ASSIGNMENT - II

Submitted by
Shabeeb Ali O.
M.Tech. Signal Processing
Semester -II
Roll No.15

April 7, 2018
Contents

0.1 Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
0.2 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
0.2.1 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
0.2.2 Multilayer Feedforward Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . 3
0.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
0.3.1 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
0.3.2 Multilayer Feedforward Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . 8
0.4 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
0.4.1 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
0.4.2 MLFFNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
0.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
0.1 Question
The dataset given to you contain 3 folders
1. linearly separable dataset
2. Nonlinearly separable dataset
3. Real world dataset

Implement classifiers using MLFFNN for all datasets and Perceptron for linearly separable dataset.

1
0.2 Theory
0.2.1 Perceptron
The perceptron occupies a special place in the historical development of neural networks: It was the first
algorithmically described neural network.The perceptron is the simplest form of a neural network used for
the classification of patterns said to be linearly separable (i.e., patterns that lie on opposite sides of a
hyperplane). Basically, it consists of a single neuron with adjustable synaptic weights and bias.

The perceptron built around a single neuron is limited to performing pattern classification with only two
classes (hypotheses). By expanding the output (computation) layer of the perceptron to include more than
one neuron, we may correspondingly perform classification with more than two classes. However, the classes
have to be linearly separable for the perceptron to work properly.

Figure 1: signal-flow graph model of perceptron

In the signal-flow graph model of figure the synaptic weights of the perceptron are denoted by w1 , w2 , ..., wm
. Correspondingly, the inputs applied to the perceptron are denoted by x1 , x2 , ..., xm . The externally applied
bias is denoted by b. From the model, we find that the hard limiter input, or induced local field, of the
neuron is
m
X
v= w i xi + b
i=1

The goal of the perceptron is to correctly classify the set of externally applied stimuli x1 , x2 , ..., xm into one
of two classes, C1 or C2 . The decision rule for the classification is to assign the point represented by the
inputs x1 , x2 , ..., xm to class C1 if the perceptron output y is +1 and to class C2 if it is -1.In the simplest
form of the perceptron, there are two decision regions separated by a hyperplane, which is defined by
m
X
v= wi xi + b = 0
i=1

2
The synaptic weights w1 , w2 , ..., wm of the perceptron can be adapted on an iteration-by-iteration basis. For
the adaptation, we may use an error-correction rule known as the perceptron convergence algorithm.

Perceptron Convergence Algorithm

Let the bias b(n) is treated as a synaptic weight driven by a fixed input equal to +1. We may thus define
the (m + 1)-by-1 input vector
x[n] = [+1, x1 (n), x2 (n), .....xm (n)]T
where n denotes the time-step in applying the algorithm. Correspondingly, we define the (m + 1)-by-1
weight vector as
w(n) = [b, w1 (n), w2 (n), .....wm (n)]T
Accordingly, the linear combiner output is written in the compact form

v(n) = w(n)T x(n)

where, in the first line, w 0 (n), corresponding to i = 0, represents the bias b. For fixed n, the equation
wT x = 0, plotted in an m-dimensional space (and for some prescribed bias) with coordinates x1 , x2 , ..., xm
, defines a hyperplane as the decision surface between two different classes of inputs.For the perceptron
to function properly, the two classes c 1 and c 2 must be linearly separable. This, in turn, means that
the patterns to be classified must be sufficiently separated from each other to ensure that the decision
surface consists of a hyperplane. This requirement is illustrated in Fig. 2 for the case of a two-dimensional
perceptron.

Figure 2: (a) A pair of linearly separable patterns. (b) A pair of non-linearly separable patterns

The algorithm for adapting the weight vector of the elementary perceptron may now be formulated as
follows:
1. If the nth member of the training set, x(n), is correctly classified by the weight vector w(n) computed
at the nth iteration of the algorithm, no correction is made to the weight vector of the perceptron in
accordance with the rule:
w(n + 1) = w(n) if wT (n)x(n) > 0 and x(n) belongs to class C1
w(n + 1) = w(n) if wT (n)x(n) ≤ 0 and x(n) belongs to class C2
2. Otherwise, the weight vector of the perceptron is updated in accordance with the rule:
w(n + 1) = w(n) − η(n)x(n) if wT (n)x(n) > 0 and x(n) belongs to class C2
w(n + 1) = w(n) + η(n)x(n) if wT (n)x(n) ≤ 0 and x(n) belongs to class C1 where the learning-rate
parameter η(n) controls the adjustment applied to the weight vector at iteration n

3
0.2.2 Multilayer Feedforward Neural Network

Figure 3: Architectural graph of a multilayer perceptron with two hidden layers.

Figure 3 shows the architectural graph of a multiplayer perceptron with two hidden layers and an output
layer. To set the stage for a description of the multilayer perceptron in its general form, the network shown
here is fully connected. This means that a neuron in any layer of the network is connected to all the neurons
(nodes) in the previous layer. Signal flow through the network progresses in a forward direction, from left
to right and on a layer-by-layer basis. The following three points highlight the basic features of multilayer
perceptrons:

• The model of each neuron in the network includes a nonlinear activation function that is differentiable.
• The network contains one or more layers that are hidden from both the input and output nodes.
• The network exhibits a high degree of connectivity, the extent of which is determined by synaptic
weights of the network.

The training proceeds in two phases:

• In the forward phase, the synaptic weights of the network are fixed and the input signal is propagated
through the network, layer by layer, until it reaches the output. Thus, in this phase, changes are
confined to the activation potentials and outputs of the neurons in the network.

• In the backward phase, an error signal is produced by comparing the output of the network with a
desired response. The resulting error signal is propagated through the network, again layer by layer,
but this time the propagation is performed in the backward direction. In this second phase, successive
adjustments are made to the synaptic weights of the network. Calculation of the adjustments for the
output layer is straightforward, but it is much more challenging for the hidden layers.

Function of the Hidden Neurons The hidden neurons act as feature detectors; as such, they play
a critical role in the operation of a multilayer perceptron.As the learning process progresses across the
multilayer perceptron, the hidden neurons begin to gradually discover the salient features that characterize

4
the training data. They do so by performing a nonlinear transformation on the input data into a new
space called the feature space. In this new space, the classes of interest in a pattern-classification task, for
example, may be more easily separated from each other than could be the case in the original input data
space. Indeed, it is the formation of this feature space through supervised learning that distinguishes the
multilayer perceptron from single layer perceptron.

MATLAB Neural Toolbox

MATLAB provides several tools for implementing neural networks for specific tasks like curve fiitng, pat-
ter classification, feedforward networks etc. Feedforward network with back propagation Algorithm can be
imlemented in MATLAB using GUI toolbox like nntools and functions like newff (obsolete) and feed-
forwardnet. The training data can be divided into train, validation and test data using divideParams
property of network object. A neural network with 9 dimensional input vector and 3 hidden layers each with
20,10,10 neurons respectively and one output vector built using newff function of MATLAB is shown below.

5
0.3 Results
0.3.1 Perceptron
Perceptron is implemented using MATlAB software and it is applied on linearly separable dataset given.
Since perceptron is a binary classifier and our dataset contains 3 classes, the classification is performed with
2 classes at a time. Hence 3 set of ouputs are showcased here.

Output for Class 1 and Class 2

Confusion Matrix
Class 1 Class 2 Class 3
Class 1 200 0 0
Class 2 0 200 0
Class 3 0 0 200

Classification Accuracy : 100%

6
Output for Class 1 and Class 3

Confusion Matrix
Class 1 Class 2 Class 3
Class 1 200 0 0
Class 2 0 200 0
Class 3 0 0 200
Classification Accuracy : 100%

Output for Class 2 and Class 3

Confusion Matrix

7
 Class 1 Class 2 Class 3
Class 1 200 0 0
Class 2 0 200 0
Class 3 0 0 200

Classification Accuracy : 100%

Observation
Effect of Learning rate parameter The learning parameter is set to a constant as η = 0.9 Since the
dataset is clearly seperable linearly the learning parameter has little effect on the speed of convergence.
Eventhough a change in speed of convergence is observable at η = 10−9 . At η = 0.9 approximate time taken
for result is 2 sec. while at η = 10−9 the time is increased approximately to 13 sec.

Effect of Bias The above results are obtained by setting bias, b = 0. When bias is changed the separating
hyperlane is moved towards the biased direction. Its effects is shown below (case of Class 1 and Class 2)

Figure 4: Classification Accuracy = 100% Figure 5: Classification Accuracy = 99.78%

As it is seen from the plots the biasing caused missclassification.

8
0.3.2 Multilayer Feedforward Neural Network
MLFFNN for Linearly Separable Dataset
The dataset contains 3 classes, Class 1, Class 2 and Class 3. All classes are of dimension 2. Since MLFFNN
are superior to single layer neural networks like perceptrons, and perceptrons are good at classification of
linearly separable data, MLFFNN gives 100% classification accuracy even with 1 neuron in hidden layer.
As the number of neurons is increased the decision boundary tends to move from linear shape to non-linear
shape. Results for 1 neuro and 5 neuron in hidden layer is shown below :
For 1 neuron in hidden layer

For 5 neurons in hidden layer

Effect of minimum gradient settings As the minimum gradient limit for the neural network to converge
is decreased the sharpness of boundary is increased

9
 Figure 6: min.grad = 0.001 Figure 7: min.grad = 0.0000001

MLFFNN for Non-linearly separable dataset - Helix

The given dataset is 3 dimensional with 2 classes.
As the number of neurons increased the classification accuracy is improved.
Since 3 dimensional data is impossible to plot with decision surface, plotting is not shown.
For 1 neuron in hidden layer
Confusion Matrix
Class 1 Class 2
Class 1 0 151
Class 2 0 151

Classification Accuracy : 50%

For 5 neuron in hidden layer

Confusion Matrix
Class 1 Class 2
Class 1 151 0
Class 2 0 151

Classification Accuracy : 100%

MLFFNN for Non-linearly separable dataset - Overlap

The dataset is 2 dimensional with 4 classes of data. The given dataset is trained for multiple hidden lay-
ers. For neural network with one hidden layer, increasing the number of neurons improves the classification
accuracy, but saturates after a certain number of neurons. After that when the number of hidden layers
is increased to two, the classification accuracy again improved and the decision surface becomes more and
more non-linear.
For one hidden layer and 2 neurons in hidden layer

10
 Figure 8: One hidden layer with 2 neurons

Confusion Matrix
Class 1 Class 2 Class 3 Class4
Class 1 53 35 3 0
Class 2 1 70 20 0
Class 3 3 27 61 0
Class 4 2 11 15 63

Classification Accuracy : 67.85%

For one hidden layer and 5 neurons in hidden layer

Figure 9: One hidden layer with 5 neurons

11
 Confusion Matrix
Class 1 Class 2 Class 3 Class4
Class 1 48 40 3 0
Class 2 0 77 13 1
Class 3 2 21 68 0
Class 4 0 14 13 64
Classification Accuracy : 70.60%

For two hidden layers and 10 neurons in hidden layer1 and 20 neurons in hidden layer 2

Figure 10: Two hidden layers with 10 neurons in hidden layer 1, 20 neurons in hidden layer 2

Confusion Matrix
Class 1 Class 2 Class 3 Class4
Class 1 70 12 9 0
Class 2 0 71 19 1
Class 3 5 8 78 0
Class 4 3 10 5 73
Classification Accuracy : 80.21%

MLFFNN for Non-linearly separable dataset - Spiral

The dataset is 2 dimensional with 4 classes of data. The given dataset is trained for multiple hidden lay-
ers. For neural network with one hidden layer, increasing the number of neurons improves the classification
accuracy, but saturates after a certain number of neurons. After that when the number of hidden layers
is increased to two, the classification accuracy again improved and the decision surface becomes more and
more non-linear.
For one hidden layer and 2 neurons in hidden layer

12
 Figure 11: One hidden layer with 2 neurons

Confusion Matrix
Class 1 Class 2
Class 1 257 14
Class 2 224 47
Classification Accuracy : 56.08%

For one hidden layer and 5 neurons in hidden layer

Figure 12: One hidden layer with 5 neurons

Confusion Matrix

13
 Class 1 Class 2
Class 1 183 88
Class 2 91 180
Classification Accuracy : 66.97%

For two hidden layers and 10 neurons in hidden layer1 and 20 neurons in hidden layer 2

Figure 13: Two hidden layers with 10 neurons in hidden layer 1, 20 neurons in hidden layer 2

Confusion Matrix
Class 1 Class 2
Class 1 259 12
Class 2 10 261
Classification Accuracy : 95.94%

MLFFNN for Real World Dataset - Glass

Real world data set contains real features of several candidates. In this dataset 9 features are given. That
is the dimension of data is 9. Number of classes is 6. For different settings and parameters the classification
accuracy does not goes beyond 55%
For 2 neurons in hidden layer
Confusion Matrix:
Class 1 Class 2 Class 3 Class 4 Class 5 Class 6
Class 1 4 11 7 0 0 0
Class 2 13 1 5 3 1 0
Class 3 0 1 5 0 0 0
Class 4 0 0 0 4 0 0
Class 5 0 0 0 1 1 1
Class 6 0 0 0 0 0 9

14
Classification Accuracy: 35.8209%

For 20 neurons in hidden layer

Confusion Matrix:
Class 1 Class 2 Class 3 Class 4 Class 5 Class 6
Class 1 17 3 2 0 0 0
Class 2 11 7 2 1 2 0
Class 3 1 1 4 0 0 0
Class 4 0 2 1 1 0 0
Class 5 0 0 0 0 2 1
Class 6 0 0 0 0 1 8

Classification Accuracy: 47.76%

MLFFNN for Real World Dataset - Jaffe

The dataset is having dimesion of 512. So the training session takes lots of time even wih samll number of
neurons in single hidden layer settings. The achieved classification accuracy is poor for 10 neurons in hidden
layer. Increasing the number of neurons does not make much difference.
For 10 neurons in hidden layer
Confusion Matrix:
Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7
Class 1 2 1 0 3 0 1 2
Class 2 1 2 2 0 0 1 1
Class 3 3 0 3 2 0 0 0
Class 4 3 0 2 1 3 1 0
Class 5 0 0 6 1 2 0 0
Class 6 3 0 0 1 2 2 1
Class 7 0 4 0 3 2 0 1

Classification Accuracy: 20.96%

15
0.4 Observation
0.4.1 Perceptron
1. As the learning rate parameter becomes nearer to zero the weight updation takes much finer steps
towards convergence.
2. As bias is changed the decision boundary biases towards the bias direction which may lead to miss-
classification

0.4.2 MLFFNN
1. As the number of neuron increases in hidden layer(s) the accuracy of classification improves but satu-
rates after a certain limit
2. The minimum gradient should be carefully chosen for each dataset to avoid overfitting of data

3. The validation data fails sometimes for randomly divided training data set.
4. If initial weight and training data division index are kept fixed the results are same (with little variance),
when the program is run again and again
5. For real worl dataset much poor classification is observed, it is may be due to complex higher dimension
of the dataset.

16
0.5 Conclusion
Perceptron shows good classification for linearly separable dataset with 2 classes.
MLFFNN shows good classification precision when the complexity of the network is increased by
increasing neurons and hidden layers. But it required much computaion power and time.

17