Maintained Activity in Neural Nets*

D. R. SMITH~ AND C. H. DAVIDSON

University of Wisconsin, Madison, Wisconsin

Abstract. Networks of cells having properties similar to those of biological neurons have
been demonstrated to be capable of supporting self-maintaining activity, using both theo-
retical and simulation techniques. Different types of steady-state and oscillatory activity
are considered and related to the network parameters: connectivity, latent summation
period and decay, refractiveness and threshold decay. It is shown that a mode of activity
where identical subsets fire repeatedly and periodically may exist in either of the above
cases even when the network is made up of elements quite widely different in properties.
The correlation of these results with certain physiological studies, and their possible func-
tion, are discussed briefly in the conclusion.

1. Introduction
Following the a p p r o a c h of R a p o p o r t [1] and Allanson [2] we consider a network
of r a n d o m l y interconnected switching elements. E a c h element fires and trans-
mits excitation or inhibition to other elements if the sum of its own incoming
excitation exceeds its threshold. I n analyses of nets of this sort it is c u s t o m a r y to
divide up the time scale, allowing firing to occur only at discrete intervals
corresponding to the p r o p a g a t i o n delay period, and thus avoid m o s t of the con-
ditional probability problems. I f these intervals are short in comparison with the
other t e m p o r a l properties of the cell, it is also hoped t h a t this procedure appro×i-
m a t e s a more continuous model.
Such an analysis leads to difference equations governing the " a c t i v i t y " of the
net (defined as the fraction of cells firing at a given t i m e interval) which are not
amenable to analytic solution [2]. I t has been suggested [1, 2], however, and
digital c o m p u t e r simulation experience of the authors has verified, t h a t a com-
m o n p r o p e r t y of such nets is a set of equilibrium values at which an approxi-
m a t e l y constant level of activity m a i n t a i n s itself. I n a s m u c h as a simple method
of analysis can be applied to these equilibria, t h e y form a convenient departure
point for study. We will start with a new general a p p r o a c h to these equilibrium
vglues.

Nomenclature
N number of cells in network
0 threshold of cells
E number of incoming excitatory connections to each cell (with randomly defined
origins)
number of incoming inhibitory connections to each cell
latent period of summation (number of consecutive time intervals over which in-
coming excitation can be summed)
* Received September, 1960; revised September, 1961.
t Present address, National Physical Laboratory, Teddington, Middlesex, England
268
 MAINTAINED ACTIVITY IN NEURAL NETS 269

"T recovery or refractory period of cell (defined so that T = 1 for a cell recovering in
one time interval)
4> instantaneous activity: equilibrium fraction of ceils firing at each time interval
+ total activity: equilibrium fraction of cells firing totalled over one refraction period
(i.e. eb = T,~).

2. Steady States
To start with, a simple net having only excitatory connections and no refrac-
toriness is considered. If the number of elements in the net is assumed large, the
probability t h a t a given cell has received i units of excitation over the preceding

The equilibrium states therefore are given by the solution or solutions of the
equation:

+=E 8 +i(i _
(1)
i~O

B y inspection, the equation is solved at ¢ = 0 for O => 1, and 6 = 1 for O =< E.

A condition which guarantees t h a t an equilibrium value is stable is t h a t

Differentiation of equation (1) shows t h a t these roots are stable for 0 ~ 2 and
0 N E - 1, respectively.
A polynomial form m a y be obtained from (1) b y means of the substitution
w = q~/(1-q~), giving

w = ,=o w, (3)

or alternatively,

1 - = o. (4)
i~l i=O

For 2 N 0 N E, equation (4) has one change of sign and therefore possesses one
and only one real positive root. This corresponds to a third equilibrium level ~bl .
T h e fact t h a t the roots q~ = 0 and 1 exist between these limits of 0 and t h a t con-
dition (2) holds for t h e m implies t h a t

dR~d4,+=+, > 1, (5)

(see Fig. l a ) . Therefore this third equilibrium level is unstable in the sense t h a t
small perturbations from it will grow monotonically until one of the other equi-
librium levels is reached. F r o m equation (4) it m a y also be shown t h a t a rise in
the value of the cell thresholds causes a rise in the value of the unstable equi-
librium point. Thus far the conclusions are equivalent to those of Rapoport.
270 D. R. SMITH AND C. H. I)AVII)SON

Tile inclusion of inhibitory connections in t.he network :raises the maximum.

number of equilibrium states. The basic equation is now:

t~ = ~ ~ 8 s d~J(1 - 4,) '<'~+~>-~--j = 4, (0),

~=0 ;=o \ * IX 2 /
The txansformation to polynomial form m a y be made as before, but gives no
simple maximum on the n u m b e r of real positive roots save t h a t of the order
s(E+I). NumericM solution has produced no total greater than four, however,
of which only two m a y be stable as before. Examination of equation (5) shows
t h a t ¢ = 0 and 4 = 1 are solutions for 0 => 1 and E - 0 ~ I respectively, and
are stable for 0 > 1 and E -- 0 > I respectively. The additional equilibrium
possibilities with the inclusion of inhibitory connections are shown in graphical
form in Figure l(b). Among these possibilities is a system with two nonzero stable
states.
Simulation of the nonrefractory systems considered above has shown an ap-
proximate agreement of experiment with t h e o r y except t h a t the experimental
activity levels tended to be a little higher than the theoretical levels. This dis-
crepancy will be further considered in the next section.
The introduction of refractory elements complicates the analysis with condi-
tional probability problems. These problems have been insufficiently appre-
ciated in the literature (see Appendix A). For s ~ T,

T T,=0 ~- 1 -- q- (1 -- ,I,)P4 (7}

where Pa is some unknown probability for the firing of the fraction(1 -- q~)
of the cells which have lain dormant one time interval or more since recovery.
Equation (7) shows t h a t in a disorganized net, an equilibrium level corre-

RE'I ,.o RI ~
/ /,5:7 /J, I
/,4"7 ,<//
1
/ q>//.',,'/;'///~
/,,///.,/,,
I ,,I,,';"7/?I',
I I I ' /~
I
I J///////#

II
00 o

(a) All excitatory connections (b) Excitatory and inhibitory connections

FIG. 1. Nonrefractory system diagrams. Equilibrium states are given by intersections
with the 45°-line.
 MAINTAINED ACTIVITY IN NEURAL NETS 27[

1.0

,9

.8

.7
• S= I
)
I'01~i~ N=IO0 E=21
,9

• q sta'elr
.7 "|
I
t~ decay to
5~
I=9

{ Excitation ~xThrseh°Id"ecayto
~xsame time ,scale
E= 5 N: time
,6 ~ Initial state
Expected level for .6 I oLk-,,
.5 ase A ( 0 = 2 ) : @=.48

.4 A
I Case B e==from 25 to 25,
0=- I
"5I
.,2
GosSe S (0.5)
.2 V similar decoy rotes as in coseA

t
0
0
yc
EXpected level for
aseLB (8=5): ~=Oj

5 I0
t
,I

IO 20 30 40 50 60 70
t

(a) Absolutely refraetm'y systems (b) Relatively refractory systems

FIG. 2. Behavior of refractory systems

sponding to ,I~ = 1 is not possible unless 0 = 0 or s => T. Pd is analytically

expressible in certain simple eases (e.g. s = 1, T = 2), and a treatment similar
to that of the preceding section gives exactly similar results except that the
upper stable equilibrium level is less than 1.
However, simulation of refractory systems shows that the levels of activity
are almost always significantly higher than those expected from equation (7).
This seems to be caused by the fact that after starting from an initial disorgan-
ized state, the activity of the network tends to "organize" itself to produce
levels of activity higher than chance. The situation is demonstrated in Figure
2(a) for a network of 512 simple identical cells. The levels of activity after the
first time interval are dose to the chance values but thereafter grow to main-
tained states considerably higher. That a similar situation prevails in networks
of more complex ceils seems to be indicated by the similar behaviors shown in
Figure 2 (b), where the refractive property is exponential and differs from cell to
cell. Figure 2 is typical of the behavior of a large number of different networks
simulated.

3. Oscillation States
The nets described in the preceding section are subject to two separate mecha-
nisms of oscillation which may support or oppose each other in any specific in-
stance. A perturbation from a steady stable state will result in oscillations about
~7'2 D.R. SMITH AND C. H. DAVIDSON

that state if

dR
-- < 0. (8)'
de
I t has been shown that such a condition cannot exist in the case of a net without
refractoriness or inhibitory connections. However, taking the inhibitory case-
first, differentiation of equation (6) (with s = 1), gives:

dR _ ~ ~ ~'÷J-'(1 - ~ ) ' ÷ ' - ' - J - ' { i + j - ~ ( E + 8)} (~)

d,# ~=0 j=0 \ 't/k3/
As I is increased, the sum (i - j) in the last term limits at 2E -- O, so t h a t
negative derivatives result. Such oscillations are necessarily simple alternations
(oscillations of period two) and have exponential bounds for perturbations small
enough so t h a t equation (6) m a y be considered a straight line at the intersection
with a 45-degree line (see Fig. 1).

S= 6

G I I I
0 I0 20 30

'L
(a) Absolutely refractory system

Z3 k Threshold decay to
17~same time scale Excitotlon
decay to
same time
.5;¢ I'.-.~"""~,,~...~_ scale
o f ' - - - r - - - - - 1 - - ' "t o t
.,4 o Io 12o 30 ~ ° Io
A N = I00
/I E-I8
.3

.2

.I

__
0o I0 20 30 40 50 60 70 80 90 100
(b) Relatively refractory system
FIG. 3. Oscillation in refractory systems
 MAINTAINED ACTIVITY IN NEURAL NETS 273

When equation (6) is considered with s > 1, differentiation should be carried

out with respect to a perturbation at the most recent time instant only. For a
perturbation from a constant state the partial derivative is a simple fraction of
the total derivative of (6), so that similar conclusions as to the initial effects
m a y be drawn. At a subsequent time, however, when the preceding s states
would not be equal, the development of a period greater than two is not pre-
cluded. Though systems having equilibrium states with such negative deriva-
tives can easily be found in practice, the oscillations resulting therefrom are not
easily separated in the corresponding simulations from the noise caused by the
finiteness of the systems. However, the observed fluctuations of activity in many
of the simulations (see for example Fig. 2a) are to a great extent simple alterna-
tions, suggesting that this mechanism is playing a part.
The second oscillation mechanism occurs in refractory systems and has been
described by Allanson [2] and Beurle [3]. Physically, the process is one of cell
exhaustion: the more cells that fire at one instant, the fewer there are available
for firing at succeeding instants, and vice versa. The oscillation period is related
to the refractive properties of the cells. Two examples of such oscillations are
shown in Fig. 3.

4. Cycling States
When a net assumes a stable state at some value of ,I~less than unity, the ques-
tion arises as to whether this state is made up of a proportion ¢ of the cells firing
at their maximum rate together with a proportion 1 - ¢ not firing at all, or
whether a larger number of cells are firing at a rate less than their maximum.
For the (inhibitory) case without refractoriness the simulations show that net
organization does not play a significant role, so that a simple analysis may be
applied. A partition of the net into proportions ~ and 1 - ¢ will be called a
maintaining partition if each of the elements of the section ¢ has 0 or more excess
of excitatory connections from its own section and each of the cells of the 1 - O
section has an excess of fewer than 0 excitatory connections from the ¢ section.
The probability of finding a maintaining partition among those of size ¢ / ( 1 - ¢ )
is
¢¢~¢(1 -- ¢) N(,-~), (10)
and the expected number of maintaining partitions of this size is

(N
N~b
) 4a*~(1 - 4)'N(1-~)' (11)

Expressions (10) and (11) have their maxima of unity at ¢ = 1 and 0, and a
minimum at ¢ = 1/2 equal to 2-N and % / 2 ~ N , respectively. Distributions (10)
and (11) are sketched in Figure 4. For a range of values of ¢ not including 0 or 1,
although the total expected number of maintained partition states m a y be size-
able their total probability is small. Starting at a random initial state, therefore,
the system m a y have to precess through a large fraction of its possible 2 v states
274 D. R. SMITH AND C. It, DAVIDSON

before reaching a maintained partition in which it will remain. Thus it would be

rare for a steady state activity to be the result of such ~ partition. This suspicion
is confirmed in simulation results for cells having short refraction properties
(s/T close to 1), where the activity involves a proporation of the elements
significantly above q,.
Thus, for stable states corresponding to values of a5 below unity, the cells are
firing at a rate less than their maximum. This nnemls that the time relations
between the firing of different cells are not preserved. On the other hand, for
stable states corresponding to a q) of unity, since the cells are firing at their maxi-
mum rate, the time relations of firing will be preserved and the same firing sets
will be reactivated at intervals equal to the refractive period. This type of ac-
tivity will be called a cycling activity. It may occur in both the oscillatory and the
steady state types of activity. Specifically, in the simulation examples of the
preceding section the systems labelled A in Figures 2 and 3 exhibited repeated
sequence activity while those labelled B did not. System 2(b)-A serves to illus-
trate that such cycling activity may still take place with cells having no absolute
refractoriness and differing relative refractory properties. Here, however, the
situation is more complex, with the cells firing an average of five times per cycle
period. The conditions for such a composite cycling activity to establish itself
are not yet clear.
Cells with refractory properties much slower than their latent sum properties
tend to be conditioned principally by the release from their blocked state.
If, in addition, the refractory property is closely similar from cell to cell, the

,0 • , '' i , ,

I ,n(da a~ 0 " 20 40 A
•8t )~I ~ \ J.(,). .(,,= o,
| I~ ~ %.. 1 ~ " " - "
IrP*
k o ,o.% ,040
~ expected numberof
maintained partitions "6pl°,., , ® "%
"' p r o b a b i l i t y of occurrence
of maintained partitions
0 20 40 (~ nit) ®

I I
/ z o 20 40 ~ I

.2 J ~ I
_L
oN |uJ FRACTIONOF INTERNAL ]
I ) 'NH'B'TORY CONNECTi'ONS (%:
N/2 N- I N oL___ ~ , . . . . . __L
~N 0 I0 20 30 ,40 50 60

FIG. 4. Maintained partitions FIG. 5. Boundary of oscillation

 MAINTAINED ACTIVITY IN NEURAL NETS 275
120

I10 (9

100

90

eO

70

60
=;
50 ° - /
40
~o
30

20
,~'~.~' FR~'I~ON OF INTERNAL
I0 INHIBITORY CONNECTIONS (~)

00 '
I0 '
20 3 '0 4 f0 5 '0 " 60

FiG. 6. Lock-in time to cycling

cycling type of activity is the dominant feature of the network. A more detailed
simulation study of such systems has shown that the conclusions obtained from
the simpler systems can be carried over to a certain extent. The systems exam-
ined were composed of 512 cells having equal absolute refractory properties and
an excitation decay such that an excitation contribution was reduced by 80 per
cent in approximately one third of the refractory period. The parameters were
such as to cause the systems to lock into cycling states in a reasonably low number
of time intervals. Figure 5 illustrates graphically the manner in which the type
of activity is related to the remaining parameters. For high proportions of
excitatory connections the networks exhibit large amplitude cell-exhaustion
oscillations. In this region the limiting value of threshold for continuing activity
is shown by the simulations to be determined by the conditions in the valleys.
Figure 6 shows that the higher the proportion of inhibitory connections, the
longer the network takes to achieve its final cycling state.

5. Conclusions
A greater understanding of the properties of random nets has been acquired
through the comparison of experiment with theory and also through the incor-
poration of inhibitory connections, not previously considered in this literature.
It is natural to compare the behaviors observed above with those observed in
the nervous matter of living organisms. Some similarity exists with the behavior
observed by Burns [4] on isolated cortical slabs, which were quiescent until once
stimulated, and then in some cases exhibited continued oscillatory activity.
276 D.R. SMITH AND C. tI. DAVIDSON

Burns' observed frequencies of about 65 cycles per second would correspond in

the model above to an equivalent absolute refractory period of 15 milliseconds,
which seems a plausible figure from physiological considerations.
Other speculation might be made abouI~ ~he possible function of the cycling
activity demonstrated here. Simulation experiments in which identical networks
were started out from a number of closely similar initiM states have shown that
the cycling states which are the final results are a more sensitive function of the
initial states than of network geometry. An estimate of the number of distinct
cycling configurations available may be made by permuting the firing times
relative to some reference element. Such an estimate gives T a'-~ using the same
nomenclature as before. Even though many of these cycles may not be possible
(in particular, those which would leave a succession of time intervals devoid of
firing events), it may still be seen that the number of possible distinct cycles may
be many times the number of cells involved. Along with this, however, it must
be remembered that information stored in a circulating cycle might not long
survive intact if small errors in the performance of the cells are cumulative [5].
All this points to a possible function as a short, term or reverberating memory
device whose purpose is to preserve information for a sufficient length of time for
it to be transferred to a more permanent store, perhaps by the mechanism of a
slow synaptic growth. Figure 6 shows that a network can be transformed from a
locked to an unlocked state either by the release of extra inhibitory (tells or by
the purging of some of the excitatory cells.
Recent work in physiology has emphasized the role of the slower dendritie
potentials, and it, has been suggested that these might have the effect of threshold
alteration. Certainly further work in both analysis and simulation of artificial
neural networks to incorporate these and other ideas is urgently called for.

A P P E N D I X A. Conditional Probability Problems

The conditional probability problems of refractory nets have often been in-
sufficiently appreciated in the literature. A cell is said to be available at time k if
it is not at that time still under the process of recove~T from a previous firing,
and is said to be hyperexcited at time k if the accumulated summed excitation on
its inputs at that time is equal to or exceeds its threshold. The question involved
is simply whether the probabilities of these two events are independent so that
they may be multiplied to obtain the probability that a cell will fire at time k.
Previously, this independence has either been implicitly assumed (Rapoport,
1952; Allanson, 1956), or such dependence as there is has been assumed negligible
for values of s near to unity. It will be shown that such assumptions are in error--
first by a theoretical contradiction, and then by a numerical example.
The following notation will be adopted for brevity:
P(A1, A2) for the joint probability that a cell is available at two suc-
cessive time instants 1 and 2.
P(H1, [I2) for the joint probability that a cell receives hyper-threshold
excitation at instants 1 and 2.
 MAINTAINED ACTIVITY IN NEURAL NETS 277

Then,
P ( A 1 , A2) ~ 0 for q, < 1
P(H1, H2) ~ 0 for 0 6 E/2 (in an all excitatory system)
P ( A 1 , A2/H1, H2) = 0 for T > 1
but
P(A1,A2,H1,H2) = P(A1,A2/H1,.lti2)'P(Hl,II~).
Therefore,
P(A1,A2,H1,H~) < P(A1,As).P(H1,H~)
for the above not uncommon conditions.
A numerical example for a system having s = 1 is instructive. For a refractory
period of two the probability Pa of (7) is approximated by the probability that
a cell has hyper-threshold stimulation at one instant but does not have hyper-
threshold stimulation at the preceding instant.

~ffio ~o . j !(E i (1 -- ~)E-,-j

For this case therefore, equation (7) becomes,

¢ = ¢ 1 -- + 2(1 -,I~)Z
i=o \ z / i=o j~o

~i! j!(E E!
@

Neglecting independence, however, equation (7) is:

The right-hand sides of the last two equations are plotted against • in Figure
7 for E = 5, 0 = 2. I t can be seen t h a t the correction for dependence has pro-

R@ I.---- - - - -
E--5
0--2
.,~ 11

..// fl
$zl
T;2 " 1

o J®
o .485 .e 1.0

, , Dependence ne~jlected
~m Dependence corrected for

FIo. 7. Effect of assumption of independence

278 D. R. S~[ITH AND C. H. DAVIDSON

duced a substantial difference in what would be the stable equilibrium state of

a disorganized net.

A P P E N D I X B. Simulation Procedures
The ten or so digital computer programs used for this study were similar in
their overall construction, and so will be described in general terms only. Each
cell of the network to be simulated is represented in the machine by one or
more stored words in the computer memory, and is assigned a numerical ad-
dress. Within the storage associated with a single cell is recorded the number of
time intervals since that cell last fired, and the addresses of all the cells at the
origin of the connections to that cell (its afferent cells). In some cases the con-
nections are designated as excitatory or inhibitory at the inputs to the cell to
correspond with the models of the preceding paper; in other cases all the out-
puts of the cells are designated as inhibitory or excitatory together to corre-
spond more closely with physiological fact. The differences in the resulting
activity turn out to be small.
A block of storage locations having been formed in this way, the program
proper searches through the network sequentially, summing the incoming
activities for each cell and testing against a threshold, recording each firing
event, and resetting the firing record of the cell when it does fire. When the
locations corresponding to the last cell of the network have been finally proc-
essed in this way, control is transferred to a loop which updates all of the firing
records by one unit, corresponding to an elapsed time interval, prints out a
record, and then reverts to the searching mode. The print-out made includes
the number of elements which fired, and some condensed format which identifies
the actual elements which fired.
To generate the network construction to some overall statistical specifica-
tions, pseudo-random number generating programs (Rotenberg, 1960; IBM,
1960) were employed to obtain lists of numbers, which could then be evenly
distributed over any desired range by multiplication by some normalizing con-
stant. Triangular distributions, obtained by adding two members of the uni-
formly distributed list, were also used.
The program proper permitted considerable variation in the characteristics
associated with the cells which determine when they would fire. Furthermore,
for both excitation summation and threshold various types of decays were
used--simple step functions, hyperbolic decay, and exponential decay--some-
times with uniform characteristics, sometimes with characteristics that varied
in a statistical fashion from cell to cell.
Most of the details of the simulations are particular to the computing ma-
chines used. The work reported here was done on the WISC [6] and on the
Burroughs 220; some was done on the IBM 704 when available.

REFERENCES
1. RAPO~'ORT,A. Ignition phenomena in random nets. Bull. Math. Biophys. (Mar. 1952), 35.
2. ALLANSON,J. T. Some properties of randomly connected neural networks. In 3rd
London Syrup. on Info. Theory I955, (Butterworths, !956), 303.
 MAINTAINED ACTINITY IN NEUTRAL NETS 279

3. BEURLE, R . L . Properties of a mass of cells capable of regenerating pulses. Phil. Trans.

lioyal Soc. B (1956), 55.
4. BURNS, B . D . Some properties of isolated cerebral cortex in the unanaesthetized cat.
J. Physiol. (1951), 156.
5. KUSHNER, H . J . On the self-organizing automata. Ph.D. Thesis, University of Wis-
consin, 1958.
6. ASMUTH, DAVIDSON, ET AL. The Wisconsin integrally synchronized computer--a
university research project. Comm. Electr. (July, 1956), 330.