Principle of Artificial Neural Networks - ToC

8868hc_9789814522731_tp.
indd 1
Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

by 41.77.16.69 on 03/25/14. For personal use only.
PRINCIPLES OF
ARTIFICIAL NEURAL
NETWORKS
3rd Edition
4/7/13 3:32 PM
ADVANCED SERIES IN CIRCUITS AND SYSTEMS

Editor-in-Charge: Wai-Kai Chen (Univ. Illinois, Chicago, USA)
Associate Editor: Dieter A. Mlynski (Univ. Karlsruhe, Germany)
Published

Vol. 1: Interval Methods for Circuit Analysis

by L. V. Kolev
Vol. 2: Network Scattering Parameters
by R. Mavaddat
Vol. 3: Principles of Artificial Neural Networks
by D Graupe
Vol. 4: Computer-Aided Design of Communication Networks
by Y-S Zhu and W K Chen
Vol. 5: Feedback Networks: Theory and Circuit Applications
by J Choma and W K Chen
Vol. 6: Principles of Artificial Neural Networks, Second Edition
by D Graupe
Vol. 7: Principles of Artificial Neural Networks, Third Edition
by D Graupe

Advanced Series in Circuits and Systems Vol. 7
PRINCIPLES OF
ARTIFICIAL NEURAL
NETWORKS
3rd Edition
Daniel Graupe
University of Illinois, Chicago, USA
World Scientific
NEW JERSEY
8868hc_9789814522731_tp.indd 2
LONDON
SINGAPORE
BEIJING
SHANGHAI
HONG KONG
TA I P E I
CHENNAI
4/7/13 3:32 PM
Published by
World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224
USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601

UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library.
Advanced Series in Circuits and Systems Vol. 7

PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS
Third Edition
Copyright 2013 by World Scientific Publishing Co. Pte. Ltd.
All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic
or mechanical, including photocopying, recording or any information storage and retrieval system now known
or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center,
Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from
the publisher.
ISBN 978-981-4522-73-1
Printed in Singapore

June 25, 2013
15:33
Principles of Articial Neural Networks (3rd Edn)
Dedicated to the memory of my parents,

to my wife Dalia,
to our children, our daughters-in-law and our grandchildren
It is also dedicated to the memory of Dr. Kate H. Kohn
ws-book975x65

June 25, 2013

15:33
This page intentionally left blank
vi
ws-book975x65
June 25, 2013
15:33
ws-book975x65

Acknowledgments
I am most thankful to Hubert Kordylewski of the Department of Electrical

Engineering and Computer Science of the University of Illinois at Chicago for his
help towards the development of LAMSTAR network of Chapter 9 of this text.
Hubert helped me with his advise in all 3 editions of this book. I am grateful to
several students who attended my classes on Neural Network at the Department of
Electrical Engineering and Computer Science of the University of Illinois at Chicago
over the past fourteen years and who allowed me to append programs they wrote as
part of homework assignments and course projects to various chapters of this book.
They are Vasanth Arunachalam, Abdulla Al-Otaibi, Giovanni Paolo Gibilisco, Sang
Lee, Maxim Kolesnikov, Hubert Kordylewski, Alvin Ng, Eric North, Maha Nujeimo,
Michele Panzeri, Silvio Rizzi, Padmagandha Sahoo, Daniele Scarpazza, Sanjeeb
Shah, Xiaoxiao Shi and Yunde Zhong.
I am deeply indebted to the memory of Dr. Kate H. Kohn of Michael Reese
Hospital, Chicago and of the College of Medicine of the University of Illinois at
Chicago and to Dr. Boris Vern of the College of Medicine of the University of
Illinois at Chicago for reviewing parts of the manuscript of this text and for their
helpful comments.
Ms. Barbara Aman and the production and editorial sta at World Scientic
Publishing Company in Singapore were extremely helpful and patient with me during all phases of preparing this book for print.
Last but not least, my sincere thanks to Steven Patt, my Editor at World Scientic Publishing Company, throughout all editions of this book, for his continuous
help and support.
vii

June 25, 2013

15:33
viii
ws-book975x65
June 25, 2013
15:33
ws-book975x65

Preface to the Third Edition
The Third Edition diers from the Second Edition in several important
aspects. I added 4 new detailed case studies describing a variety of applications,
(new Sections 6.D, 7.C, 9.C, 9.D) together with their respective source codes. This
brings the total number of application to 19. This will allow the reader rst-hand
access to a wide range of dierent concrete APPLICATIONS of Neural Networks
ranging from medicine to constellation detection, thus establishing the main claim
of Neural Networks, namely the claim of its GENERALITY.
The new case studies include an application to a non-linear prediction problems
(case study 9.C), which are indeed a eld where articial neural networks are and
will play a major role. This case study also compares performances of two dierent neural networks in terms of accuracy and computational time, for the specic
problem of that case study. Also, two new Section (9.6 and 9.8) were added to
Chapter 9.
Text organization is also modied. The Chapter on the Large Memory Storage
and Retrieval Neural Network (LAMSTAR) is moved from Chapter 13 of the Second Edition, to become Chapter 9 in the present Edition. Consequently, the old
Chapter 9 to 12 are now Chapters 10 to 13. This allows teaching and self-study to
follow the main Articial Neural Networks (ANN) in a more logical order in terms
of basic principles and generality. We consider that in short courses, Chapters 1 to
9 will thus become the core of a course on ANN.
It is hoped that these text and this enhanced Edition can serve to show and
to persuade scientists, engineers and program developers in areas ranging from
medicine to nance and beyond, of the value and the power of ANN in problems
that are ill-dened, highly non-linear, stochastic and of time-varying dynamics and
which often appear to be beyond solution.
Additional corrections and minor modications are also included, as are other
updates based on recent developments including those relating to the authors
research.
Daniel Graupe
Chicago, IL
March 2013
ix

June 25, 2013

15:33
ws-book975x65
June 25, 2013
15:33
ws-book975x65

Preface to the Second Edition
The Second Edition contains certain changes and additions to the First
Edition. Apart from corrections of typos and insertion of minor additional details
that I considered to be helpful to the reader, I decided to interchange the order of
Chapters 4 and 5 and to rewrite Chapter 13 so as to make it easier to apply the
LAMSTAR neural network to practical applications. I also moved the Case Study
6.D to become Case Study 4.A, since it is essentially a Perceptron solution.
I consider the Case Studies important to a reader who wishes to see a concrete
application of the neural networks considered in the text, including a complete
source code for that particular application with explanations on organizing that application. Therefore, I replaced some of the older Case Studies with new ones with
more detail and using most current coding languages (MATLAB, Java, C++). To
allow better comparison between the various neural network architectures regarding
performance, robustness and programming eort, all Chapters dealing with major
networks have a Case Study to solve the same problem, namely, character recognition. Consequently, the Case studies 5.A (previously, 4.A, since the order of these
chapters is interchanged), 6.A (previously, 6.C), 7.A, 8.A, have all been replaced
with new and more detailed Case Studies, all on character recognition in a 6 6
grid. Case Studies on the same problem have been added to Chapter 9, 12 and
13 as Case Studies 9.A, 12.A and 13.A (the old Case Studies 9.A and 13.A now
became 9.B and 13.B). Also, a Case Study 7.B on applying the Hopeld Network to
the well known Traveling Salesman Problem (TSP) was added to Chapter 7. Other
Case Studies remained as in the First Edition.
I hope that these updates will add to the readers ability to better understand
what Neural Networks can do, how they are applied and what the dierences are
between the dierent major architectures. I feel that this and the case studies with
their source codes and the respective code-design details will help to ll a gap in the
literature available to a graduate student or to an advanced undergraduate Senior
who is interested to study articial neural networks or to apply them.
Above all, the text should enable the reader to grasp the very broad range of
problems to which neural networks are applicable, especially those that defy analysis
and/or are very complex, such as in medicine or nance. It (and its Case Studies)
xi
June 25, 2013
15:33
xii
ws-book975x65
Principles of Artificial and Neural Networks
should also help the reader to understand that this is both doable and rather easily
programmable and executable.

Daniel Graupe
Chicago, IL
September 2006
June 25, 2013
15:33
ws-book975x65

Preface to the First Edition
This book evolved from the lecture notes of a rst-year graduate course entitled
Neural Networks which I taught at the Department of Electrical Engineering
and Computer Science of the University of Illinois at Chicago over the years 1990
1996. Whereas that course was a rst-year graduate course, several Senior-Year
undergraduate students from dierent engineering departments, attended it with
little diculty. It was mainly for historical and scheduling reasons that the course
was a graduate course, since no such course existed in our program of studies and in
the curricula of most U.S. universities in the Senior Year Undergraduate program. I
therefore consider this book, which closely follows these lecture notes, to be suitable
for such undergraduate students. Furthermore, it should be applicable to students
at that level from essentially every science and engineering University department.
Its prerequisites are the mathematical fundamentals in terms of some linear algebra
and calculus, and computational programming skills (not limited to a particular
programming language) that all such students possess.
Indeed, I strongly believe that Neural Networks are a eld of both intellectual
interest and practical value to all such students and young professionals. Articial
neural networks not only provide an understanding into an important computational architecture and methodology, but they also provide an understanding (very
simplied, of course) of the mechanism of the biological neural network.
Neural networks were until recently considered as a toy by many computer
engineers and business executives. This was probably somewhat justied in the
past, since neural nets could at best apply to small memories that were analyzable
just as successfully by other computational tools. I believe (and I tried in the
later chapters below to give some demonstration to support this belief) that neural
networks are indeed a valid, and presently, the only ecient tool, to deal with very
large memories.
The beauty of such nets is that they can allow and will in the near-future allow,
for instance, a computer user to overcome slight errors in representation, in programming (missing a trivial but essential command such as a period or any other
symbol or character) and yet have the computer execute the command. This will
obviously require a neural network buer between the keyboard and the main proxiii
June 25, 2013
15:33

xiv
ws-book975x65
grams. It should allow browsing through the Internet with both fun and eciency.
Advances in VLSI realizations of neural networks should allow in the coming years
many concrete applications in control, communications and medical devices, including in articial limbs and organs and in neural prostheses, such as neuromuscular
stimulation aids in certain paralysis situations.
For me as a teacher, it was remarkable to see how students with no background
in signal processing or pattern recognition could easily, a few weeks (1015 hours)
into the course, solve speech recognition, character identication and parameter
estimation problems as in the case studies included in the text. Such computational
capabilities make it clear to me that the merit in the neural network tool is huge.
In any other class, students might need to spend many more hours in performing
such tasks and will spend so much more computing time. Note that my students
used only PCs for these tasks (for simulating all the networks concerned). Since
the building blocks of neural nets are so simple, this becomes possible. And this
simplicity is the main feature of neural networks: A house y does not, to the
best of my knowledge, use advanced calculus to recognize a pattern (food, danger),
nor does its CNS computer work in picosecond-cycle times. Researches into neural
networks try, therefore, to nd out why this is so. This leads and led to neural
network theory and development, and is the guiding light to be followed in this
exciting eld.
Daniel Graupe
Chicago, IL
January 1997
June 25, 2013
15:33
ws-book975x65

Contents
Acknowledgments
vii
Preface to the Third Edition
ix
Preface to the Second Edition
xi
Preface to the First Edition
xiii
Chapter 1. Introduction and Role of Articial Neural Networks
Chapter 2.
Fundamentals of Biological Neural Networks
Chapter 3. Basic Principles of ANNs and Their Early Structures

3.1.
3.2.
3.3.
3.4.
Basic Principles of ANN Design . . . . . .

Basic Network Structures . . . . . . . . .
The Perceptrons Input-Output Principles
The Adaline (ALC) . . . . . . . . . . . .
.
.
.
.
.
.
.
.
9
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
10
11
13
The Basic Structure . . . . . . . . . . . . . . . . . . .

The Single-Layer Representation Problem . . . . . . .
The Limitations of the Single-Layer Perceptron . . . .
Many-Layer Perceptrons . . . . . . . . . . . . . . . . .
Perceptron Case Study: Identifying Autoregressive
Parameters of a Signal (AR Time Series Identication)
.
.
.
.
.
.
.
.
17
22
22
24
. .
25
Chapter 4. The Perceptron

4.1.
4.2.
4.3.
4.4.
4.A.
17
Chapter 5. The Madaline
37
5.1. Madaline Training . . . . . . . . . . . . . . . . . . . . . .

5.A. Madaline Case Study: Character Recognition . . . . . . .
xv
37
39
June 25, 2013
15:33
xvi
ws-book975x65
Chapter 6.

Back Propagation
59
6.1. The Back Propagation Learning Procedure . . . . . . . . 59

6.2. Derivation of the BP Algorithm . . . . . . . . . . . . . . . 59
6.3. Modied BP Algorithms . . . . . . . . . . . . . . . . . . . 63
6.A. Back Propagation Case Study: Character Recognition . . 65
6.B. Back Propagation Case Study: The Exclusive-OR (XOR)
Problem (2-Layer BP) . . . . . . . . . . . . . . . . . . . . 76
6.C. Back Propagation Case Study: The XOR Problem
3 Layer BP Network . . . . . . . . . . . . . . . . . . . . . 94
6.D. Average Monthly High and Low Temperature Prediction
Using Backpropagation Neural Networks . . . . . . . . . . 112
Chapter 7. Hopeld Networks
7.1.
7.2.
7.3.
Introduction . . . . . . . . . . . . . . . . . . . . . . . .
Binary Hopeld Networks . . . . . . . . . . . . . . . .
Setting of Weights in Hopeld Nets Bidirectional
Associative Memory (BAM) Principle . . . . . . . . .
7.4. Walsh Functions . . . . . . . . . . . . . . . . . . . . .
7.5. Network Stability . . . . . . . . . . . . . . . . . . . . .
7.6. Summary of the Procedure for Implementing the
Hopeld Network . . . . . . . . . . . . . . . . . . . . .
7.7. Continuous Hopeld Models . . . . . . . . . . . . . . .
7.8. The Continuous Energy (Lyapunov) Function . . . . .
7.A. Hopeld Network Case Study: Character Recognition
7.B. Hopeld Network Case Study: Traveling Salesman
Problem . . . . . . . . . . . . . . . . . . . . . . . . . .
7.C. Cell Shape Detection Using Neural Networks . . . . .
123
. . 123
. . 123
. . 125
. . 127
. . 129
.
.
.
.
.
.
.
.
. . 147
. . 170
Chapter 8. Counter Propagation

8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . .
8.2. Kohonen Self-Organizing Map (SOM) Layer . . . . . .
8.3. Grossberg Layer . . . . . . . . . . . . . . . . . . . . .
8.4. Training of the Kohonen Layer . . . . . . . . . . . . .
8.5. Training of Grossberg Layers . . . . . . . . . . . . . .
8.6. The Combined Counter Propagation Network . . . . .
8.A. Counter Propagation Network Case Study: Character
Recognition . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 9.
Large Scale Memory Storage and Retrieval (LAMSTAR)

Network
9.1.
9.2.
131
132
133
135
185
.
.
.
.
.
.
.
.
.
.
.
.
185
186
186
187
189
190
. . 190
203
Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Basic Principles of the LAMSTAR Neural Network . . . . 204
June 25, 2013
15:33
ws-book975x65
Contents

9.3.
9.4.
9.5.
9.6.
9.7.
9.8.
9.9.
9.A.
Detailed Outline of the LAMSTAR Network . . . . . .

Forgetting Feature . . . . . . . . . . . . . . . . . . . .
Training vs. Operational Runs . . . . . . . . . . . . .
Operation in Face of Missing Data . . . . . . . . . . .
Advanced Data Analysis Capabilities . . . . . . . . . .
Modied Version: Normalized Weights . . . . . . . . .
Concluding Comments and Discussion of Applicability
LAMSTAR Network Case Study: Character
Recognition . . . . . . . . . . . . . . . . . . . . . . . .
9.B. Application to Medical Diagnosis Problems . . . . . .
9.C. Predicting Price Movement in Market Microstructure
via LAMSTAR . . . . . . . . . . . . . . . . . . . . . .
9.D. Constellation Recognition . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . 240
. . 253
275
10.1.
10.2.
10.3.
10.4.
10.5.
10.6.
Motivation . . . . . . . . . . . . . . . . . . . . . . .
The ART Network Structure . . . . . . . . . . . . .
Setting-Up of the ART Network . . . . . . . . . . .
Network Operation . . . . . . . . . . . . . . . . . . .
Properties of ART . . . . . . . . . . . . . . . . . . .
Discussion and General Comments on ART-I and
ART-II . . . . . . . . . . . . . . . . . . . . . . . . .
10.A. ART-I Network Case Study: Character Recognition
10.B. ART-I Case Study: Speech Recognition . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
305
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Chapter 12. Statistical Training

12.1.
12.2.
12.3.
12.4.
275
275
279
280
281
. . . 283
. . . 283
. . . 297
Chapter 11. The Cognitron and the Neocognitron

Background of the Cognitron . . . . .
The Basic Principles of the Cognitron
Network Operation . . . . . . . . . . .
Cognitrons Network Training . . . . .
The Neocognitron . . . . . . . . . . .
205
211
213
213
214
217
218
. . 220
. . 236
Chapter 10. Adaptive Resonance Theory
11.1.
11.2.
11.3.
11.4.
11.5.
xvii
Fundamental Philosophy . . . . . . . . . . . . . . . . .
Annealing Methods . . . . . . . . . . . . . . . . . . . .
Simulated Annealing by Boltzman Training of Weights
Stochastic Determination of Magnitude of Weight
Change . . . . . . . . . . . . . . . . . . . . . . . . . .
12.5. Temperature-Equivalent Setting . . . . . . . . . . . . .
12.6. Cauchy Training of Neural Network . . . . . . . . . .
12.A. Statistical Training Case Study: A Stochastic Hopeld
Network for Character Recognition . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
305
305
305
307
309
311
. . 311
. . 312
. . 312
. . 313
. . 313
. . 314
. . 315
June 25, 2013
15:33
xviii
ws-book975x65
12.B. Statistical Training Case Study: Identifying AR Signal

Parameters with a Stochastic Perceptron Model . . . . . . 318

Chapter 13. Recurrent (Time Cycling) Back Propagation Networks

13.1. Recurrent/Discrete Time Networks . . . . . . . . . .
13.2. Fully Recurrent Networks . . . . . . . . . . . . . . .
13.3. Continuously Recurrent Back Propagation Networks
13.A. Recurrent Back Propagation Case Study: Character
Recognition . . . . . . . . . . . . . . . . . . . . . . .
327
. . . 327
. . . 328
. . . 330
. . . 330
Problems
343
References
349
Author Index
357
Subject Index
361

Principle of Artificial Neural Networks - ToC

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Principle of Artificial Neural Networks - ToC

Transféré par

Droits d'auteur :

Formats disponibles

8868hc_9789814522731_tp.

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

ADVANCED SERIES IN CIRCUITS AND SYSTEMS

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Vol. 1: Interval Methods for Circuit Analysis

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Advanced Series in Circuits and Systems Vol. 7

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

British Library Cataloguing-in-Publication Data

Advanced Series in Circuits and Systems Vol. 7

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Dedicated to the memory of my parents,

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

June 25, 2013

This page intentionally left blank

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

I am most thankful to Hubert Kordylewski of the Department of Electrical

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

June 25, 2013

This page intentionally left blank

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Preface to the Third Edition

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

June 25, 2013

This page intentionally left blank

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Preface to the Second Edition

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial and Neural Networks

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Preface to the First Edition

June 25, 2013

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial and Neural Networks

June 25, 2013

Principles of Articial Neural Networks (3rd Edn)

Principles of Artificial Neural Networks Downloaded from www.worldscientific.com

Preface to the Third Edition

Preface to the Second Edition

Preface to the First Edition

Chapter 1. Introduction and Role of Articial Neural Networks

Fundamentals of Biological Neural Networks

Chapter 3. Basic Principles of ANNs and Their Early Structures

Basic Principles of ANN Design . . . . . .

The Basic Structure . . . . . . . . . . . . . . . . . . .

Chapter 4. The Perceptron

Chapter 5. The Madaline

5.1. Madaline Training . . . . . . . . . . . . . . . . . . . . . .

June 25, 2013