ASSOCIATIVE MEMORY NETWORKS

INTRODUCTION
Enable to retrieve a piece of data from only a tiny sample itself
Also called as Content Addressable Memories (CAM)
Contrast to Address-addressable Memory.
It has set of patterns as memories.
Stored patterns must be unique (i.e) different patterns in each location.
If the same pattern exists more than one location in the CAM, address is found to be
ambiguous (having more than one meaning).
Hamming Distance is the number of mismatched components of x & x vectors.
x=(x1,x2,..xn) x=(x1,x2,..xn)
Architecture may be feed forward or iterative.
Two types of memory
Auto associative memory
Hetero associative memory

Training algorithms for pattern association

 Two algorithms
Hebb rule
Outer Product rule
HEBB RULE
Used for finding the weights of an associative memory neural net.
Training vector pairs are denoted by s:t
Algorithm
Step 0: Initialize weights to 0
wij=0 (i= 1 to n, j=1 to M)
For each training target input output vectors s:t, perform steps 2-4.
Activate the input layers for training input.
xi=si(for i=1 to n)
Activate the output layers for target output.
yi=ti(for j=1 to m
Start weight adjustment.
wij(new)=wij(old)+x iyj (for i=1 to n, j=1 to m)
Used with patters that can be represented as either binary or bipolar vectors.

OUTER PRODUCT RULE

Alternative method for finding weight of an associative net.
Input=> s=(S ,...S ,.,S )
1 i n
(t ,...t ,.,t )
Output =>t= 1 j m
T
Outer product of two vectors is the product of matrices S=s and T=t, between
[n x 1] matrix and [1xm] matrix. The
transpose is taken for input matrix given.
 Auto associative memory network
Theory
Training input and target output vectors are same.
Determination of weight is called storing of vectors
Weight is set to zero.
Auto associative net with no self connection.
Increases net ability to generalize.
Architecture
The Fig gives the architecture of Auto associative memory network.
Input and target output vectors are same.
The input vector has n inputs and output vector has n outputs.
The input and output are connected through weighted connections.

Training Algorithm
Step 0: Initialize weights to 0
wij=0 (i= 1 to n, j=1 to m)
For each of the vector that has to be stored, perform steps 2-4.
Activate the input layers for training input.
xi=si(for i=1 to n)
Activate the output layers for target output.
yi=si(for j=1 to n)
 Adjust weight .
wij(new)=wij(old)+x iyj
Weight can also be found by using

Testing Algorithm
An auto associative network can be used to determine whether the given vector
is a known or unknown vector.
A net is said to recognize a known vector if the net produces a pattern of
activation on the output which is same as one stored.
Testing procedure is as follows:
Step 0: Set weights obtained from Hebbs rule
Step 1: For each testing input vector perform steps 2 to 4
 Step 2: Activation of inputs is equal to input vector.
Step 3: calculate the net input for each output unit j=1 to n:
Step 4: Calculate the output by applying the activation vector over the net input

Hetero associative memory network

Theory
The training input and target output vectors are different.
Determination of weight is by Hebb rule or Delta rule.
Input has n units and output has m units and there is a weighted
interconnection between input and output.
Architecture
The architecture is given in the Fig.
 Testing Algorithm
Step 0: Initialize the weights from the training algorithm.
Step 1: Perform steps 2-4 for each input vector presented.
Step 1: Set the activation inputs equal to current input vector

(j=1 to m)
Determine the activation of the output units
The output vector y is obtained gives the pattern associated with the input
vector x.
If the responses are binary then the activation function will be as

Bidirectional associative memory(BAM)]

Theory
Developed by Kosko in the year 1988.
Performs backward & forward search.
Encodes binary/bipolar pattern using Hebbian learning rule
Two types
Discrete BAM
 Continuous BAM
Architecture
Weights are bidirectional
X layer has n input units
Y layer has m output units.
T.
Weight matrix from X to Y is W and from Y to X is W

Discrete bi directional auto associative memory

Here weight is found to be the sum of outer product of bipolar form.
Activation function is defined with nonzero threshold.
Determination of weights
Input vectors is denoted by s() and output vector as t().Then the weight
matrix is denoted by
s()=(S1 (),...Si (),.,Sn ())
Output =>t ()= (t1 (),...tj (),.,tm ())
Weight matrix is determined using the Hebb Rule.
If the input vectors is binary, then weight matrix W={wij}=

If the input vectors are bipolar, the weight matrix W={wij}=

Weight matrix will be in bipolar form, neither the input vectors are binary
or not.

ACTIVATION FUNCTION FOR BAM

 The Activation Function is based on whether the input target vector pairs
used are binary or bipolar.
The Activation function for Y layer with binary input vectors is

with bipolar input vector is

The activation function for the X layer with binary input vector is

With bipolar input vector is

If threshold value is equal to the net input, then the previous output value
is calculated is left as the activation of that unit. Signals are sent only from
one layer to the other and not in both directions.

Testing algorithm for discrete bam

Test the noisy patterns entering into the network.
Testing algorithm for the net is as follows:
Step 0: Initialize the weights to store vectors. Also initialize all the
activations to zero.
Step 1:Perform steps 2-6 for each testing input.
Step 2: Set the Activation of X layer to current input
patterns, presenting the input x to X layer and
presenting the input pattern y to Y layer. It is
bidirectional memory.
Step 3: Perform steps 4-6 when the activations are not converged.
Step 4: Update the activation of units in Y layer. Calculate the net input.

Applying the Activation, we get

yj=f(yinj).
Send this signal to X layer.
Step 5: Update the activation of units in X layer.

Calculate the net input

 Applying the activation over the net input
xi=f(xini)
Send this signal to Y layer.
Step 6: Test for convergence of the net. The
convergence occurs if the activation vectors x and y reach equilibrium. If
this occurs, then stop, else continue.

CONTINUOUS BAM
It uses logistic Sigmoid function as the activation functions for all units.
It may be binary sigmoid or bipolar sigmoid.
Bipolar sigmoid function with high gain, converge to vector state and acts
like DBAM.
If the input vectors are binary, s(), t(), the weights are determined using
the formula
wij=
If a binary logistic function is used, then the activation function is

If the activation function is bipolar logistic function then,

Net input calculated with bias is included

ANALYSIS OF HAMMING DISTANCE, ENERGY FUNCTION AND STORAGE CAPACITY

Hamming distance-number of mismatched components of two given

bipolar/binary vectors.
Denoted by H [X, X]
Average distance =[1/n]
X = [ 1 0 1 0 1 1 0]
Y = [ 1 1 1 1 0 0 1]
HD = 5
Average = 5/7.
Stability is determined by Lyapunov function (Energy function).
 Change in energy
Memory capacity min(m,n)
n is the number of units in X layer and m is the number of units in the

Y layer.