8 vues

Transféré par naeemkashif

Neural Networks by Muhammad Amjad

- IJAIEM-2014-02-27-059
- Obstacle Avoidance in Mobile Robot Using Neural Network---IEEE2011
- NEURAL NETWORKS AND BACK PROPAGATION ALGORITHM
- An Improved Method for Synchronous Control of Complex Multi-motor System5
- MACROSCOPIC QUANTUM EFFECTS IN BIOPHYSICS AND CONSCIOUSNESS
- Recognition of Meetei Mayek Script Using Swarm Optimization and Neural Network
- APIStructures_RPStruct
- OFF-LINE Signature Verification Using Neural Network Approach
- 3
- Gasm Manual
- Python Exercises
- 122.pdf
- DSA_Ex00
- ic_1403_-neural_networks_and_fuzzy_logic_control
- An Intelligent Approach for Dos Attacks Detection
- 36030544 2002 Using Neural Networks in the Static Evaluation Function of a Computer Chess Program Robe
- Reis Filho Francisco a., Senger Eduardo C.-protection of Transmission Lines With Series Compensation New Tools
- Dev Guide Unicode Conversion
- Marc Strickert- Self-Organizing Neural Networks for Sequence Processing
- Neural Network complete

Vous êtes sur la page 1sur 82

Introduction

Suppose we want to predict tomorrows stock markets

average y(t+1) based on todays economic indicators xi(t)

Suppose we want the prediction to be

based on todays and yesterdays

economic indicators xi(t) & xi(t-1)

In this case we can augment the number

of inputs

RECURRENT NETWORKS

Introduction

However, if we wish the network to consider an arbitrarysized window of time in the past (e.g. sometimes two days,

sometimes 20 days), then a different solution is required

The recurrent networks provide one such solution

They are used to learn sequential or time-varying patterns,

in which the current value of the pattern is dependent upon

its past history

RECURRENT NETWORKS

Architecture

A specific group of

units receives feedback

signals

from

the

previous time step

These units are known

as context units

The weights on the feedback connections to the context

units are fixed (e.g. 1)

RECURRENT NETWORKS

Architecture

Context unit is feedforward connected to its

hidden unit & all other

hidden unit

Number of context units

= number of hidden

units

One hidden unit has feedback connection to one context

unit only

RECURRENT NETWORKS

Training of Weights

At time t, the activations of the context units are the

activations (output signals) of the hidden units at the

previous time step

The weights from the context units to the hidden units are

trained in exactly the same manner as the weights from the

input units to the hidden units

Thus, at any time step, the training algorithm is the same as

for standard backpropagation

RECURRENT NETWORKS

Utilization

After training, the recurrent

nets can be presented an

arbitrary sized sequence and

it will predict the next element

of the sequence

It can also be trained to

classify or label a sequence.

We give an arbitrary sized

portion of the sequence as

input and the network will try

to tell us the name of the

sequence

RECURRENT NETWORKS

Example: Next valid letter in a string of characters

Suppose that a string of characters are generated by a small

finite-state grammar. The string begins with the symbol B

and ends with the symbol E

At each decision point,

either path can be taken

with equal probability

Two examples of the

possible strings are

B PVVE

&

BTX XTTVPS E

RECURRENT NETWORKS

Example: Architecture used

RECURRENT NETWORKS

Example: Training

The training patterns for the neural net consisted of 60,000

randomly generated strings

The string length ranged from 3 to 30 letters (plus the Begin

and End symbols at the beginning and end of strings)

The letters in the string are presented sequentially to the net,

one by one, respecting the order in which they appear

Each letter in the string becomes an input, on its turn, and its

successor is considered as the output

RECURRENT NETWORKS

Example: Training

The training algorithm for the context sensitive grammar is:

RECURRENT NETWORKS

Example: Training

String

BTXSE

RECURRENT NETWORKS

Example: Testing & Utilization

After training, we hope that the net has learnt the grammar

Given a valid sequence of letters, it can be used to predict the

next valid letter in the sequence

An interesting application can be the determination of a

given string as a valid string

As each symbol is presented, the net predicts the possible

valid successors of that symbol (output units with activations

of 0.3 or more are counted as valid successors)

RECURRENT NETWORKS

Example: Testing & Utilization

If the next letter in the string is not one of the predicted valid

successors, the string is rejected. If all the letters pass the

test, then the string is accepted as valid

The reported result for 70,000 random strings, 0.3% (210) of

which were valid, are that the net correctly reject 99.7% of

the invalid strings and accepted 100% of the valid strings

The net also performed perfectly on a set of extremely long

strings (100 or more characters in length)

RECURRENT NETWORKS

Example:

Utilization

Testing

&

such that we can establish

context with the help of two

samples

If the context is deeper than

more complex network may

be required

RECURRENT NETWORKS

Example:

Utilization

Testing

&

adapted to other problems

For example, letters may be

replaced

by

features

composing the letters

or

by words

sentence

composing

RECURRENT NETWORKS

Conclusion

The architecture that we have studied is called simple

recurrent network as it contains a single hidden layer

It is also called Elman network and is similar to an

architecture proposed by Jordan

It is implemented in Matlabs neural network tool-box

RECURRENT NETWORKS

References

Read Section 7.2.3, page 372 of Laurene Fausett

ART-1

Unsupervised method of clustering

Allows increase in number of clusters only if required

Developed by Carpenter & Grossberg in 1987

ART-1 is designed for clustering binary valued vectors and

ART-2 is for continuous valued inputs

ART-1

Architecture

ART-1

Architecture

The architecture of the computational units for ART1

consists of

F1 units (input and interface units),

F2 units (cluster units),

and a reset unit

The reset unit implements user control over the degree of

similarity of patterns placed on the same cluster

ART-1

Architecture

Each unit in the F1(a) input layer is connected only to the

corresponding unit in the F1(b) interface layer.

Each unit in the input and interface layers is connected to

the reset unit. The reset unit is connected to every F2

(cluster) unit.

ART-1

Architecture

Each unit of the interface layer is connected to each unit in

the cluster layer by two weighted pathways.

The interface unit Xi is connected to the cluster unit Yj by

bottom up weight bij.

Similarly, unit Yj is connected to unit Xi by top-down

weights tji

The cluster layer is a competitive layer in which only the

uninhibited node with the largest net input has a non-zero

activation

ART-1

Training

Notations

n

m

maximum number of clusters that can be

formed

bij

bottom-up weights (from interface unit Xi to

cluster unit Yj)

tji

binary top-down weights (from cluster unit Yj

to interface unit Xi)

p

vigilance parameter

s

binary input vector (an n-tuple)

x

activation vector for interface layer (binary)

x norm of vector x, defined as the sum of the

components xi

ART-1

Training: Competition-to-Learn Phase

A binary input vector s is presented to the input layer, and

the signals are sent to the corresponding X units. These

interface units then broadcast to the cluster layer over

connection pathways with bottom-up weights

Each cluster unit computes its net input and the units

compete for the right to be active

The unit with the largest net input sets its activation to 1; all

others have an activation of zero. Let the index of the

winning unit be J. This winning unit becomes the candidate

to learn the input pattern

ART-1

Training: Similarity Check

A signal is then sent down from cluster to interface layer

(multiplied by the top-down weights)

The interface X units remain on only if they receive nonzero signals from both the input and cluster units.

The activation vector of interface layer has the states of

individual units as its components. Those units are 1 or on

which have non-zero inputs

The norm of the vector x (the activation vector for the

interface layer) gives the number of components in which the

top-down weight vector for the winning cluster unit t J and

the input vector s are both 1. This quantity is sometimes

referred to as the match.

ART-1

If the ratio of x to s is greater than or equal to the

vigilance parameter, the weights (top down and bottom up)

for the winning cluster unit are adjusted

The use of the ratio allows an ART1 net to respond to

relative differences. This reflects the fact that a difference of

one component in vectors with only a few non-zero

components is much more significant than a difference of one

component in vectors with many non-zero components

ART-1

Training: Search for another unit

if 1st candidate is rejected

However, if the ratio is less than the vigilance parameter, the

candidate unit is rejected, and another candidate unit must

be chosen

The winning cluster unit becomes inhibited, & it cannot be

chosen again as a candidate on this learning trial, & the

activations of the input & interface units are reset to zero

The same input vector again sends its signal to the interface

units, which again send this as the bottom-up signal to the

cluster layer, & the competition is repeated (but without the

participation of any inhibited units)

ART-1

Training: Search for another unit

if 1st candidate is rejected

The process continues until either a satisfactory match is

found (a candidate is accepted) or all units are inhibited

The action to be taken if all units are inhibited must be

specified by the user: reduce the value of the vigilance

parameter thus allowing less similar patterns to be placed on

the same cluster, or to increase the number of cluster units,

or simply to designate the current input pattern as an outlier

that could not be clustered

ART-1

Training: Search for another unit

if 1st candidate is rejected

At the end of each presentation of a pattern, all cluster units

are returned to inactive status and are available to

participate in the next competition

ART-1

Training

ART-1

Training

ART-1

Training

ART-1

Training

ART-1

Architecture

ART-1

Training: Example

The ART1 algorithm is used to group 4 vectors in at most 3

clusters

Vectors are: [1

[0

[1

[0

1

0

0

0

0

0

0

1

0]

1]

0]

1]

ART-1

Training: Example

ART-1

Training: Example

ART-1

Training: Example

ART-1

Training: Example

ART-1

Training: Example

ART-1

References

Read

Section 5.2 of Laurene Fausette

ART-1

References

Read

Section 5.2 of Laurene Fausette

RBF NETWORKS

Architecture

RBF NETWORKS

Architecture

Radial Basis Function Networks consist of:

- A hidden layer of k Gaussian neurons

- A set of weights wi

For Gaussian basis functions, the activation is given as

RBF NETWORKS

Training

RBFNs are trained using a two step process:

Determine the K neurons using:

Kohonen Unsupervised Learning

Determine the weights

Supervised Learning (Backpropagation)

RBF NETWORKS

Architecture

RBF NETWORKS

Architecture

RBF NETWORKS

independent of other outputs: the network can be

viewed as m independent networks

RBF NETWORKS

Outputs as

linear

combination of

hidden layer of

RBF neurons

Inputs

RBF NETWORKS

References

Engelbrecht

Chapter 5

HOPFIELD NETWORK

Architecture

HOPFIELD NETWORK

Architecture

HOPFIELD NETWORK

Utilization

HOPFIELD NETWORK

Utilization

HOPFIELD NETWORK

Utilization

HOPFIELD NETWORK

Training by Hebbs Rule

HOPFIELD NETWORK

Training by Hebbs Rule

pattern to be stored

If there is a positive weight between them, then each would

reinforce the other (each one is making a positive

contribution to the activation of other)

For (0, 1) or (1, 0) activations, negative weights reinforce

each others activations

HOPFIELD NETWORK

Training by Hebbs Rule

It gives a positive weight, if both activations are (1, 1) or (0,

0) and a negative weight if they are (1, 0) or (0, 1)

HOPFIELD NETWORK

Training by Hebbs Rule: Bi-polar activations

The formula gives positive weights for (1, 1) and negative

weights for (-1, 1) activations

HOPFIELD NETWORK

Storage Capacity

HOPFIELD NETWORK

References

Laurene Fausette

Kevin Gurney

Section 3.4.4

Chapter 5 & 6

BOLTZMANN MACHINE

Introduced in 1983 by Hinton & Sejnowski

The architecture of a Boltzmann machine consists of a set of

units and a set of bidirectional connections between pairs of

units

Not all units are connected, however if two units are

connected then their connection is bidirectional and the

weights in both directions are same, i.e. wij = wji

A unit may also have a self-connection with weight wii

BOLTZMANN MACHINE

Architecture

Boltzmann machine to

solve

the

Travelling

Salesman problem

Units are arranged in a

two dimensional array

The units within each row

are fully interconnected

Similarly, the units within

each column are fully

interconnected

BOLTZMANN MACHINE

Architecture

Architecture for 10 city TSP

BOLTZMANN MACHINE

Architecture

The state xi of a unit Xi is either on (1) or off (0)

The objective of the neural net is to maximize to consensus

function over all the units of the net

BOLTZMANN MACHINE

Architecture

The net attempts to find this maximum by letting each unit

attempt to change its state from on to off or vice versa

The change in consensus for a change of state of unit Xi is

The coefficient [1 2xi] will be +1 if unit Xi is currently off

and -1 if it is on

BOLTZMANN MACHINE

Architecture

However, unit Xi does not necessarily change its state, even if

doing so would increase the consensus of the net

We update is done probabilistically so that the chances of the

net getting stuck in a local maximum are reduced

BOLTZMANN MACHINE

Architecture

The probability of the net accepting a change in state is

Lower values of T makes it more likely that the net will accept

a change of state that increases its consensus and less likely

that it will accept a change that reduces its consensus

The control parameter T is gradually reduced as the net

search proceeds

BOLTZMANN MACHINE

Architecture

The connection pattern for the net is as follows

BOLTZMANN MACHINE

Architecure

BOLTZMANN MACHINE

Architecture

Consider the relation between the penalty weight -p and the

bonus weight b

Allowing a unit Uij to turn on should be encouraged only if no

other units are on in that column or row (no city can be

visited twice, neither can be two different cities visited at the

same time)

If we set p > b, then our objective will be achieved, and if a

unit turns on with a unit already on in its row or column,

then the consensus function will have a negative change

BOLTZMANN MACHINE

Architecture

Now let us consider the relation between the constraint weight

b and the distance weights

Let d denote the maximum distance between any two cities on

the tour

Suppose that no unit in on in a column j and in row i

Allowing the Uij to turn on should be encouraged and the

weights should be set so that the consensus will be increased if

it turns on

BOLTZMANN MACHINE

Architecture

The change in consensus will be b di,k1 di,k2

where k1 indicates city visited at stage j-1 of the tour, and k2

denotes the city visited at stage j+1 (and the city i is visited at

stage j)

BOLTZMANN MACHINE

Architecture

Since d is the maximum possible distance, hence the minimum

change in consensus b di,k1 di,k2 will be equal to b 2d

Since the change in consensus should be positive we can take

the value of b > 2d for the net to function properly

BOLTZMANN MACHINE

Setting the weights

The weights for a Boltzmann machine are set manually

Their values are chosen such that the net will tend to make

transitions towards a maximum of the consensus function

BOLTZMANN MACHINE

Algorithm

BOLTZMANN MACHINE

Algorithm

BOLTZMANN MACHINE

Algorithm: Initial Temperature

The initial temperature should be taken large enough so that

the probability of accepting a change of state is approximately

0.5, regardless of whether the change is beneficial or

detrimental

BOLTZMANN MACHINE

Example

The Boltzmann machine was used to solve the TSP

The starting configuration had approximately half the units

on

The parameters were: To = 20, b = 60, & p = 70

The cooling schedule was Tnew = 0.9Told after each epoch

An epoch consisted of each unit attempting to change its

value

BOLTZMANN MACHINE

Example

The experiment was repeated 100 times for different initial

configurations

Typically valid tours were produced in 10 or fewer epochs &

for all 100 configurations valid tours were found in 20 or

fewer epochs

In these experiments, it was rare for the net to change its

configuration once a valid tour was found

BOLTZMANN MACHINE

Example

Five tours of less length were

BOLTZMANN MACHINE

References

Laurene Fausette

Section 7.1.1

- IJAIEM-2014-02-27-059Transféré parAnonymous vQrJlEN
- Obstacle Avoidance in Mobile Robot Using Neural Network---IEEE2011Transféré parSVSEMBEDDED
- NEURAL NETWORKS AND BACK PROPAGATION ALGORITHMTransféré parIninteligible
- An Improved Method for Synchronous Control of Complex Multi-motor System5Transféré parMohamed Berriri
- MACROSCOPIC QUANTUM EFFECTS IN BIOPHYSICS AND CONSCIOUSNESSTransféré parničeom izazvan
- Recognition of Meetei Mayek Script Using Swarm Optimization and Neural NetworkTransféré parChetan Dasgupta
- APIStructures_RPStructTransféré parVilciu Constantin
- OFF-LINE Signature Verification Using Neural Network ApproachTransféré parseventhsensegroup
- 3Transféré parHedra Girgis
- Gasm ManualTransféré parAbhijeet Sa
- Python ExercisesTransféré parronak142
- 122.pdfTransféré parAjverson
- DSA_Ex00Transféré parFaysal Hossain
- ic_1403_-neural_networks_and_fuzzy_logic_controlTransféré parPrabu Lakshmanan
- An Intelligent Approach for Dos Attacks DetectionTransféré parAmrYassin
- 36030544 2002 Using Neural Networks in the Static Evaluation Function of a Computer Chess Program RobeTransféré parbrisas2k
- Reis Filho Francisco a., Senger Eduardo C.-protection of Transmission Lines With Series Compensation New ToolsTransféré parOmar Chayña Velásquez
- Dev Guide Unicode ConversionTransféré paralourenco
- Marc Strickert- Self-Organizing Neural Networks for Sequence ProcessingTransféré parRoots999
- Neural Network completeTransféré parDeepthi Katragadda
- bpj lesson 16Transféré parapi-294682711
- lab5-stringspointerstoolsTransféré pardnfkjn
- Basic Java Programs (1)Transféré paranil
- nov23Transféré parRohitashv_Kaw_7349
- UpdateTransféré parWanM.Syamim
- WCE2008_pp228-233Transféré parcsrajmohan2924
- 1Transféré parAntriksh Bahuguna
- C_assignments.pdfTransféré parsuneet
- Neural NetsTransféré parArioston Júnior
- 14915Transféré parthyagosmesme

- Formal Methods - Lecture03Transféré parnaeemkashif
- Ch1 SlidesTransféré parRahat
- Three Level Architecture of DBMSTransféré parShokin Ali
- HEC Guidelines of Credit SharingTransféré parnaeemkashif
- Python GuideTransféré parnaeemkashif
- Formal Methods - Lecture 02Transféré parnaeemkashif
- Neural Networks- IntroductionTransféré parnaeemkashif
- SMSCaster E-Marketer User ManualTransféré parDidi Junaedi
- Icmap Ml 303Transféré parnaeemkashif
- Neural Networks - Lecture 03.pptTransféré parnaeemkashif
- Formal Methods - Lecture01.pdfTransféré parnaeemkashif
- RDIT OutlineTransféré parnaeemkashif

- Introduction to the Theory of Neural ComputationTransféré parJon Arnold Grey
- Calicuit Sem 5Transféré parSuresh Kumar
- Adaptation Interaction and Its Application to NNTransféré parMuhammad Zulfiqar
- SP 561Transféré parkishore_k777
- HWCRTransféré parNikhil Sai Nethi
- A Neural Network Approach for Early Cost Estimation of Structural Systems of BuildingsTransféré parHai Dai Gia
- Decision Support System Using Decision Tree and Neural NetworksTransféré parAlexander Decker
- Automotive Idle Speed Control SystemTransféré parIvan Fauzi Ryanto
- Plain Woven Fabric Defect Detection Based on Image Processing and Artificial Neural NetworksTransféré parseventhsensegroup
- Wind power forecasting: A Case Study in Terrain using Artificial IntelligenceTransféré parIRJET Journal
- Applying Data Mining Techniques to Breast Cancer AnalysisTransféré parLuis Rei
- Matconvnet Manual jjjTransféré parLidija Antic
- Visual Speech Analysis,Transféré parlbkft
- Artificial Neural NetworksTransféré parAnup Shokhwal
- schaul10a.pdfTransféré parBisma Nusa
- A Neural Network Model and Algorithm for the Hybrid Flow Shop Scheduling Problem in a Dynamic EnvironmentTransféré parmeysam64
- _lec_2_-_neural_network_perceptron_adaline_.pdfTransféré parLoris Franchi
- cheatsheet_luckycallorTransféré parmmmax
- Daubechies Wavelet Tool for Multi Resolution Face RecognitionTransféré parChetan
- Malaysian Journal of Computer ScienceTransféré parAadil Muklhtar
- ANN and Power SystemTransféré par032457
- 1-s2.0-S0895717713000290-mainTransféré parNishtha Agarwal
- Neural NetworksTransféré parGoutham Beesetty
- IEEE Xplore Full-Text PDF_ (1)Transféré parPawan Mangal
- A RubaaiTransféré parsherykhann
- Prediction of Aerodynamic Coefficients for Wind Tunnel Data Using a Genetic Algorithm Optimized Neural NetworkTransféré parGrace Thompson
- ImplementationTransféré parTamilSelvan
- Neural Networks and Deep Learning Chap3Transféré parMohit Duseja
- miah-2016-ijais-451501Transféré parDino Omanović
- Seminarie3_1Transféré parRenata Smith