Neural Comput & Applic (2009) 18:377–380
DOI 10.1007/s00521-008-0188-0
ORIGINAL ARTICLE
Classification error of multilayer perceptron neural networks
Lihua Feng Æ Weihu Hong
Received: 24 May 2007 / Accepted: 10 April 2008 / Published online: 1 May 2008

Springer-Verlag London Limited 2008
Abstract In subject classification, artificial neural net-
works (ANNS) are efficient and objective classification
methods. Thus, they have been successfully applied to the
numerous classification fields. Sometimes, however, clas-
sifications do not match the real world, and are subjected to
errors. These problems are caused by the nature of ANNS.
We discuss these on multilayer perceptron neural networks.
By studying of these problems, it helps us to have a better
understanding on its classification.
Keywords Artificial neural networks

Multilayer perceptron neural networks
Subject

Class
Classification
Error
1 Introduction
In subject classification, artificial neural networks (ANNS
for short) are efficient and objective classification methods.
Since 90s, hundreds of papers have been published about
different types of classifications, such as the classifications of
economic systems, remote sensing images, communication
signals, the failure of rocket engines, energy levels, Chinese
medicine, fingerprint identification, environmental qualities,
water sources, stabilities of rock, and ship noises [1–4].
However, some examples have indicated that there are still
some problems with some of the ANNS classification. In this
L. Feng (&)
Department of Geography, Zhejiang Normal University,
No. 688 Yingbin Road, Jinhua 321004, China
e-mail: fenglh@zjnu.cn
W. Hong
Department of Mathematics, Clayton State University,
Morrow, GA 30260, USA
paper, these problems are discussed on the multilayer per-
ceptron neural networks (MLPNNS) [5, 6].
2 Method and fundamental theory of multilayer
perceptron neural networks
MLPNNS are modeled on the human brain and consist of a
number of artificial neurons. MLPNNS are self-organizing,
adaptive, learning and fault tolerance [7]. MLPNNS can be
used to imitate human brain activities [8, 9]. MLPNNS
have parallel distributed information processing structure
in which nonlinear regression is applied to input/output
mappings. Without employing any mathematical model,
MLPNNS learn by past experiences and process those
nonlinear, noisy, or imprecise data through simulation,
memorization and correlation. The subject classification is
performed by self-adaptation method of pattern recognition
[10].
The major MLPNNS algorithms include: Hebb’s rule,
Delta rule, Kohenon’s learning law, and BP algorithm
[6, 11]. BP algorithm, a back propagation algorithm, is
developed by Rumelhart et al., the PDP Group, in 1985. It
realized the concept of multilayer networks proposed by
Minsky. As a typical feedback, each of MLPNNS consists
of an input layer, an output layer and at least one hidden
layer. Figure 1 is the topological structure.
The neural network with error back propagation algo-
rithm is called BP network [12] whose learning process
includes two phases: forward phase, and backward phase. In
forward phase, by Sigmoid function ½f ð xÞ ¼ 1=ð1 þ ex Þ;
signal propagates forward layer by layer. The status of
neural cell at one layer can influence only the status of
neural cell at next layer. If output signal is not as desired,
changing the weights of neural cell at each layer, and at the
123
378 Neural Comput & Applic (2009) 18:377–380

The error was calculated through backward propaga-

tion (m = m, m - 1, …, 1).
6. Update the weights and thresholds and propagate them
backward, layer by layer:
Wijm ðt þ 1Þ ¼ Wijm ðtÞ þ gdm j yi
m1
h i
þ a Wijm ðtÞ  Wijm ðt  1Þ ð5Þ
h i
hm
j ð t þ 1 Þ ¼ h m
j ð t Þ þ gd m
j þ a h m
j ð t Þ  h m
j ð t  1 Þ ð6Þ

where t is the iteration, g is the learning-rate parameter

[g [ (0, 1)], and a is the momentum constant [a [ (0, 1)].
7. Go back to step 2, repeat the process 2–7 until the
selected error criterion is satisfied
XX
2 
E¼ Tjk  ymj 2 ð7Þ
k j

Fig. 1 The topological structure of a simple three-layer-feed-forward After the network training, all weights and thresholds are
neural network
determined. Thus, it can start the classification.

same time propagating backward the errors associated with For simplicity, Table 1 lists five typical samples S
output neurons. Error is calculated recursively until the (about equal to criteria of five classes) each with eight
selected error criterion is satisfied. attributes xi. The criteria of Table 1 are used as five
Assume an m layer neural network. ym j is the output of learning samples. For the input variables, the desired out-
the jth node at the mth layer, while y0j = xj is the input of puts of BP network are: 0.1, 0.3, 0.5, 0.7, and 0.9. The
the jth node. Let Wm ij be the weight between yi
m-1
and ym
j , network would therefore have 8 input nodes, 1 output node,
m and 12 hidden nodes. Thus, the topological structure of BP
and hj be the threshold of the jth node at the mth layer. The
BP network training can be described as follows [13]: network is (8, 12, 1).
In order to speed up the calculation, the original data
1. Set all the weights and thresholds to random numbers should be normalized:
between -1 and 1.
2. Chose a pair of number (xk, Tk) from training data as x0i ¼ ðxi  xmin Þ=ðxmax  xmin Þ ð8Þ
the input variables, put these input variables to the where xmax and xmin are the maximum and minimum values
input layer (m = 0), and let of the attributes, respectively. Hence, x0i is within [0, 1].
y0i ¼ xki ; for all nodes i ð1Þ The normalized data is used as the input data of the BP
network, and the training parameters are selected for training
where k is the layer number. and learning processes. Here, the learning-rate parameter g is
3. Signal propagates forward through network by Eq. (2) 0.6, and the momentum parameter a is 0.5. After 100,000
which calculates the output ym j of the jth node from the trainings and learnings, the total error of network
first layer to the last one. E = 0.0002. Thus, the network minimizes error to the
!

 X desired threshold. The correspondent outputs of the network
m
ymj ¼ F sj
m
¼F Wijm ym1
i þ hj ð2Þ are 0.11, 0.30, 0.50, 0.70, and 0.89. Using the median of these
i
outputs as classification criteria, we get five classes as shown
where F(s) is Sigmoid function. in Table 2.
4. Calculate the error of node at the output layer:


 Table 1 Five typical samples S each with attributes xi
dm
j ¼ y m
j 1  y m
j T k
j  y m
j ð3Þ Samples x1 x2 x3 x4 x5 x6 x7 x8

The error is the difference between the real output and S1 2 2 2 2 2 2 2 2

the desired value. S2 4 4 4 4 4 4 4 4
5. Calculate the error of the previous layer: S3 6 6 6 6 6 6 6 6

X S4 8 8 8 8 8 8 8 8
0 m1
dm1
j ¼ F s j Wijm dm
i ð4Þ S5 10 10 10 10 10 10 10 10
i

123
Neural Comput & Applic (2009) 18:377–380
Table 2 The five classes based
Number 1 2
on the outputs
Desired outputs 0.1
Network outputs 0.11
Demarcation points [0, 0.20]
Classes I II
Since the trained network has already simulated and
memorized the relationship between the input and the
output variables, it can be used for subject classification.
Giving a set of data as the input data, through well trained
network, and after that, comparing with classification cri-
teria, the input can be told which class it belongs to.
3 The classification problem
The calculation of real data shows that some problems exist
using a well trained MLPNNS:
1. The change of positions of attributes in input data
causes the possible classification changes. Let s1 and s2
be two samples with the same attributes:
s1 ¼ ð4; 4; 4; 4; 10; 10; 10; 10Þ
s2 ¼ ð10; 10; 10; 10; 4; 4; 4; 4Þ
The only difference between s1 and s2 is the position
change between x1 * x4 and x5 * x8. But the outputs
of BP networks are: y1 = 0.46 (belongs to class III),
y2 = 0.74 (belongs to class IV). The results indicate that
for the same subject, the position change of its attributes
causes the classification change.
2. For those samples with large values of attributes, BP
network losses its ability of classification. Let s3 be a
sample with some big attributes:
s3 ¼ ð2; 2; 2; 100; 100; 100; 100; 100Þ
In this sample, five attributes are as big as 100. The result
of BP network is y3 = 0.29 (belongs to class II), it can be
seen that the classification is not true.
3. The difference of the number of nodes in hidden layer
causes the difference of classification. At present, there
is no theoretical justification about the number of
nodes in hidden layer [14], thus, in practice the choice
of the number of nodes in hidden layer is kind of
arbitrary. The outputs from the network show that a
different number of nodes in hidden layer correspon-
dents to a different classification. For example,
s4 ¼ ð2; 2; 2; 2; 8; 8; 8; 8Þ
When the number of nodes in hidden layer, n = 4,
y4 = 0.48 which belongs to class III while when n = 10,
y4 = 0.31 which belongs to class II.
379
3 4 5
0.3 0.5 0.7 0.9
0.30 0.50 0.70 0.89
(0.20, 0.40] (0.40, 0.60] (0.60, 0.80] (0.80, 1.00]
III IV V
4. The change of learning rate causes the change of
classification. The bigger the learning rate g is, the
faster the weights update. General speaking, g can be
big if only the calculation is stable [14]. But the results
show that, when g changes, sometimes, the classifica-
tion changes. For the previous s1, when g = 0.1,
y1 = 0.44, which belongs to class III; when
g = 0.000001, y1 = 0.62, which belongs to class IV.
5. The change of momentum parameter causes the
change of classification. In calculation, the momentum
parameter a is adjusted according to the experience
[14]. The results show that when a changes, classifi-
cation changes accordingly. For the same s2, when
a = 0.8, y2 = 0.84, which belongs to class V; when
a = 0.1, y2 = 0.73, which belongs to class IV.
6. The different number of learning times causes the
different classification. The outputs of BP network
show that when the learning times T is different, the
classification is different. For instance, let s5 be:
s5 ¼ ð5; 5; 5; 5; 5; 5; 5; 5Þ
When T = 10,000, y5 = 0.37, which belongs to class II;
when T = 100,000, y5 = 0.44, which belongs to class III.
7. The different criteria cause the different classification.
At present, there is no common rule about the criterion,
it is chosen by one’s experience. The calculation shows
that with different criteria, classification would be
different. For example, for the sample s5, when
E \ 0.05, y5 = 0.47, which belongs to class III; when
E \ 0.005, y5 = 0.38, which belongs to class II.
8. It is difficult for BP network to discriminate sharply.
For instance, let s7 and s8 be:
s7 ¼ ð6; 4; 4; 4; 4; 4; 4; 4Þ
s8 ¼ ð60; 4; 4; 4; 4; 4; 4; 4Þ
From general knowledge, s8 should belong to a higher class
than that s7 belongs to. The outputs of BP network, how-
ever, show that they belong to the same class: y7 = 0.37
(class II), y8 = 0.37 (class II). Thus we can see that the
discrimination ability of BP network is limited.
9. The bigger the attribute’s value is, the lower the class
to that it belongs. For instance, let s9 and s10 be two
different samples:
s9 ¼ ð10; 5; 2; 2; 2; 2; 2; 2Þ
s10 ¼ ð30; 5; 2; 2; 2; 2; 2; 2Þ
123
380
In s9, x1 = 10, while in s10, x1 = 30, from this we can see
that the class that s10 belongs to is higher than the class that
s9 belongs to. But the outputs of BP network are: y9 = 0.25
(class II), y10 = 0.16 (class I).
4 Conclusions
The advantage of MLPNNS is that there is no need to devise a
mathematical model in order to perform a specific task.
MLPNNS process information through interconnected pro-
cessing elements (neurons). By nonlinear fitting, MLPNNS
realize the mapping between input and output. MLPNNS
have the advantages of self-learning, self-organization,
self-adaptation, and fault tolerance. Thus, MLPNNS have
been successfully applied to pattern recognition and system
identification. However, according to our experience,
MLPNNS do not always comply with real situations, and
sometimes, they can cause large errors with testing samples.
These problems are caused by the nature of MLPNNS. By
studying of these problems, it helps us to have a better
understanding on MLPNNS classification and find a way to
improve their performance.
Acknowledgments This work was supported by National Natural
Science Foundation of China (No. 40771044) and Zhejiang Provincial
Science and Technology Foundation of China (No. 2006C23066). We
would like to thank the editor and reviewers for their comments that
improved the article.
123
Neural Comput & Applic (2009) 18:377–380
References
1. McCulloch WS, Pitts W (1943) A logical calculus of the ideas
immanent in nervous activity. Bull Math Biophys 5:115–137
2. Minsky M, Papert SA (1987) Perceptrons-expanded edition: an
introduction to computational geometry. MIT Press, Cambridge
3. Zhang ZH, Lu YC, Zhang P (1999) Discovering classification
rules by using the neural networks. Chin J Comput 22(1):108–112
4. Li Q, Wang ZZ (2000) Remote sensing information classification
based on artificial neural network and knowledge. Acta Autom
Sin 26(2):233–239
5. Callan R (1994) The essence of neural networks. Prentice Hall,
Englewood Cliffs
6. Coppin B (2004) Artificial intelligence illuminated. Jones and
Bartlett Publishers, Sudbury
7. Kohonen T (2000) Self-organizing maps. Springer, New York
8. Chen TP (1994) Approximation problems in system identification
with neural networks. Sci China (Ser A) 24(1):1–7
9. Lippmann RP (1987) An introduction to computing with neural
nets. IEEE ASSP Mag 4(2):4–22
10. Bishop C M (1996) Neural networks for pattern recognition.
Oxford University Press, New York
11. Hebb DO (1949) The organisation of behavior: a neuropsycho-
logical theory. Lawrence Erlbaum Assoc, New Jersey
12. Gurney K (1997) An introduction to neural networks. UCLA
Press, Los Angels
13. Zhou JC, Zhou QS, Han PY (1993) Artificial neural network—
the implementation of the 6th computer. Publishing House of
Science Popularization, Beijing, pp 47–51
14. Xu LN (1999) Neural network control. Publishing House of
Haerbin Industry University, Haerbin, pp 15–16