Custom VLSI implementations of neural networks

Early attempts

Analog VLSI circuits for spiking neural networks

Giacomo Indiveri
Neuromorphic Cognitive Systems group Institute of Neuroinformatics University of Zurich and ETH Zurich

The idea of making custom analog VLSI implementations of neural networks dates back to the late 80s - early 90s:
[Holler et al. 1989, Satyanarayana et al. 1992, Hammerstrom 1993, Vittoz 1996]

General purpose computing Full-custom analog implementation Neural network accelerator PC-boards

Competing with Intel steamroller Communication - bandwidth limited Difcult to program

September 17, 2009

Current research
Technological progress Power-dissipation/computational power Application-specic focus Embedded system integration
3 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

1 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

Silicon neuron designs

Many VLSI models of spiking neurons have been developed in the past, and many are still being actively investigated:

Silicon neural network characteristics

Above threshold (strong inversion) Mixed analog/digital Rate-based Real-time Conductance-based Large-scale, event-based networks

Below threshold (weak inversion) Fully analog Spiking Accelerated-time Integrate-and-Fire Small-scale, hard-wired

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

4 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

5 / 78

Most designs can be traced back to one of two types of silicon neurons,

Why subthreshold neuromorphic VLSI

Exploit the physics of silicon to reproduce the bio-physics of neural systems.

MOSFETs in subthreshold

Vd Vg Ids Vs
where I0 denotes the nFET current-scaling parameter n-FET subthreshold transfer function Ids = I0 en Vg /UT eVs /UT eVd /UT

n denotes the nFET subthreshold slope factor

UT the thermal voltage Vg the gate voltage, Vs the source voltage, and Vd the drain voltage. The current is dened to be positive if it ows from the drain to the source
G.Indiveri (NCS @ INI) ICANN09 Tutorial 6 / 78 G.Indiveri (NCS @ INI) ICANN09 Tutorial 9 / 78

Diffusion and saturation

Exponential voltage dependence

Subthreshold n-FET

Vg Vs Ir Vs Ir
Qs

Vd If Vg Vd If
Qd

Ids = I0 en Vg /UT eVs /UT eVd /UT is equivalent to: Ids = Ids = I0 e
V Ug U s T V T

10 10 10

-2

-3

subthreshold

-4

I0 e Ir

U g Ud
T

10 10

-5

T
-6

Ids (A)

If

10 10 10 10 10 10

-7

above threshold

If Vds > 4UT the Ir term becomes negligible, and the transistor is said to operate in the saturation regime: Ids = I0 en Vg /UT Vs /UT

-8

-9

-10

-11

-12

0.5

1.5

2.5

3.5

4.5

Vgs (V)

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

10 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

11 / 78

n-FETs and p-FETs

In Complementary Metal-Oxide Semiconductor (CMOS) technology, there are two types of MOSFETs: n-FETs and p-FETs

One, two, and three transistor circuits

Ideal current source Current-mirror
I in I out

Vd

Vs

Vg Vs

Vg Vd

Vb

Vin

I out
M1 M2

Vd

Inverting amplier

Differential pair
I1 V1 I2 V2

In traditional CMOS circuits, all n-FETs have the common bulk potential (Vb ) connected to Ground (Gnd), and all p-FETs have a common bulk potential (typically) connected to the power supply rail (Vdd ). The corresponding (complementary) equation for the p-FET is Ids = I0 e
G.Indiveri (NCS @ INI)

Vin

Vout
Vbn

M2

Vs M1

M3

p (Vdd Vg )/UT

(Vdd Vs )/UT

(Vdd Vd )/UT

ICANN09 Tutorial

12 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

14 / 78

The differential-pair
V1 Vs
UT

The transconductance amplier

I1 V1 I2 V2
V2 + Vb Iout V out

I1 = I0 e I2 = I0 e

Vdd
M4

Vdd
M5

V1

V2 Vs
UT

M2

Vs M1

M3

Ib = I1 + I2 = I0 e UT e
V Us T

Vb

Vbn

Iout I1 I2 Vs Ib Vb
M3 M2

Vout V2

Iout = Ib tanh

2UT

(V1 V2 )

Ib I0
V1

1 e UT + e UT
V1 V2
1 0.8

x 10

V1
I1

M1

In the linear region (|V1 V2 | < 200mV ): Iout gm (V1 V2 ) where gm = Ib 2UT

I2

I1 = Ib e I2 = Ib

e UT
V1
UT

+e
V1

V2
UT

I1, I2 (A)

0.6

0.4

e UT e UT + e UT
V1 V2

0.2

0 0.3

0.2

0.1

0 V1V2 (V)

0.1

0.2

0.3

is a tunable conductance.

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

15 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

16 / 78

What is a synapse?

Synapses in the nervous system

In 1897 Charles Sherrington introduced the term synapse to describe the specialized structure at the zone of contact between neurons as the point in which one neuron communicates with another.

Eex (Na+, ...) Glutammate

Electrical | Chemical
Vmem Gl GABA Einh (K+, Cl, ...) Cmem

Excitatory | Inhibitory Depressing | Facilitating AMPA | NMDA ...

2005 winner of the Science and Engineering Visualization Challenge. by G. Johnson, Medical Media, Boulder, CO.

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

19 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

20 / 78

Synaptic transmission

EPSC and EPSP

730 O. SACCHI, M. L. ROSSI, R. CANELLA, AND R. FESCE

In chemical synapses the presynaptic and postsynaptic membranes are spearated by extracellular space. The arrival of a presynaptic action potential triggers the release of neurotransmitter in the extracellular space. The neurotransmitters react with the postsynaptic receptors and depolarize the cell. Chemical synaptic transmission is characterized by specic temporal dynamics.
FIG .

maximal preganglionic stimulation. Superimposed Superimposed excitatory EPSCs recorded in a sympathetic neuron at different membrane potentials between 010 and NATURE OF THE POSTSYNAPTIC RECEPTOR ACTIVATED BY 0100 mV (in 10 mV steps) under 2-electrode voltage-clamp conditions. post-synaptic currents (EPSCs) s intervals, to NATURAL AC . To correctly model and reproduce evoked Excitatory post-synaptic potential Cell was held at 050 mV and then repeatedly stepped, at 20 it was important whether or membrane levels at observed. Note presence recorded 10/ 0which synaptic current wasCa concentration was 5 synaptic currents,response toto determinethe nicotinic (EPSP) in multiple 30 neuron at different of fast I in 0 in a mV tracings. External not conductances other than those associated to mM. receptor were activated by nerve-released ACh. In fact, ACh membrane potentials (from Sacchi et pre-synaptic spikes terminals is thought to genrelease by the presynaptic nerve(from Nicholls et al. cell was estimated in culture, broad series addition al., 1998). for the rat sympathetic neuron putative M- erate in the ganglion asustaining theof side effects, intransmis1992). whereas the activation-deactivation time constants for to the central role of fast excitatory
H
2/ Na

2.

Typical synaptic currents at ganglionic synapse evoked by supra-

low the linear t. This might be partly because of inward rectication by the ACh-evoked current (Fieber and Adams 1991; Mathie et al. 1990), but it is at least partly accounted for by the changes in ionic gradients that are associated with even small ionic ows (especially accumulation of K / ions in the perineuronal space) (Belluzzi and Sacchi 1990). In all neurons the decay of EPSC was well t by a single exponential function. The average time constant, t, was 7.5 ms at 075 mV (n 14) and was scarcely voltage-dependent: it decreased exponentially with a constant of 260.4 mV from 0105 to 025 mV (Fig. 3A). EPSC amplitudes decreased in the average (5 neurons) to 69% of the initial value when Ca 2/ concentration in the bath was lowered from the usual 5 to 2 mM; the current decay time constants were also somewhat decreased (by 21% in the average). The voltage dependence of the decay time constant and the estimated reversal potential did not appear to be affected by calcium concentration (Fig. 3B).

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

21 / 78

sion. Autoreceptors were proposed to control evoked transmitter release (see Starke et al. 1989, for a review). The evidence for a physiological role in the ganglion seems to R E S U L G.Indiveri (NCS @ INI) TS ICANN09 be greatest for the muscarinic autoreceptors, which are pos-/ 78 Tutorial 22 tulated to mediate a negative feedback loop that reduces Synaptic current at the sympathetic ganglionic neuron transmitter mobilization (Koelle 1961; Koketsu and Yamada

currents were 220 ms in the 060/ 030 mV voltage range (Owen et al. 1990).

Neural network models

VLSI synapses in classical neural networks

The role of the VLSI synapse in implementations of classical neural network models is that of a multiplier. Multiplying synaptic circuits have been implemented using a wide range of analog circuits, ranging from the single MOS-FETs to the Gilbert multiplier.
Iin1 I1 I2 Iin2

In classical neural network theory signals are (tipically) continuous values that represent the neurons mean ring rate, neurons implement a saturating non-linearity transfer function (S) on the inputs weighted sum, the synapse implements a multiplication between the neurons input signal (Xi ) and its corresponding synaptic weight (wi ).
G.Indiveri (NCS @ INI) ICANN09 Tutorial 24 / 78

Ib

Figure: Schematic of half of a Gilbert multiplier. This circuit multiplies Iin1 and Iin2 by Ib if Iin1 + Iin2 = Ib .
G.Indiveri (NCS @ INI) ICANN09 Tutorial 26 / 78

VLSI synapses in pulse-based neural networks

Vdd

Linear pulse integrators

A linear integrator is a linear low-pass lter. Its impulse response should be a decaying exponential. With VLSI and subthreshold MOSFETS its fairly easy to implement exponential voltage to current conversion, and linear voltage increase or decrease over time.
Vdd

Wi

IWi Vmem Cmem

Vi =

IWi Cmem

In pulse-based neural networks the weighted contribution of a synapse can be implemented using a single transistor. In this case p-FETs implement excitatory synapse, and n-FETs implement inhibitory synapses. The synaptic weight can be set by changing the Wi bias voltage or the t duration.

Vg

Id
Vg(t)

Vdd

Vc C

Id =

d C dt Vc

Id (t ) = I0 e UT
Id(t)

(Vdd Vg (t ))

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

27 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

28 / 78

Linear charge-and-discharge integrator

Vw
UT

Log-domain pulse integrator

Csyn UT Iw = I0 e UT
V M I Vw Mw Isyn Iw Mpre Csyn Vsyn Msyn

Iw = I0 e
V M I Vw Csyn Vsyn Msyn Isyn

, c , d

(Vsyn Vw )
UT

I
Csyn UT

I = I0 e Ic = C

(Vdd V )
UT

I = I0 e Ic = C

(Vdd V )

d dt

d dt

(Vdd Vsyn )
(Vdd Vsyn )
UT

Mw Iw Mpre

(Vdd Vsyn )
(Vdd Vsyn )
UT

Isyn = I0 e d dt

Isyn = I0 e Isyn (t ) =

+ (t ti ) I e c
syn
+ + (t ti ) d I e

Isyn = Isyn Csyn UT

d
UT dt

Vsyn

(charge phase) (discharge phase) c c + d

1 d dt d dt Isyn + Isyn = I0 Iw0 I

syn

Isyn Iw
(Vw Vdd )
UT

Isyn (t ) = I0 e

t c f c(c +d ) t d

, with f =

n t

, f<

Isyn + Isyn =

, Iw0 = I0 e

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

29 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

30 / 78

The diff-pair integrator (DPI)

Iw = I0 e
Msyn Isyn
Vw
UT

DPI equations
DPI transfer function: d Iout + Iout dt 1 Iout
Iout Ig

M I Iin

Csyn Vsyn

I = I0 e Ic = C

(Vdd V )
UT

I 1 + Ig I Iin ,

Iin

Vthr

d dt

Mthr

Min

(Vdd Vsyn )
Vsyn

d dt

Iout + Iout

if Iout

Ig , Iw

Vw

Mw Iw Mpre

Iin = Iw e

e
Vsyn
UT

UT

Response to t pulse at time ti :

Vthr
UT

+e
UT

Iout (ti +t ) =

Ig Iin I

1 e

+ Iout (ti )e , Iout (ti ) = Iout (ti 1 +t )e :

ti ti 1

Isyn = I0 e d dt d dt Isyn + Isyn

(Vdd Vsyn )

Response to an arbitrary spike train (t ) = i (t ti ), with t d dt Vsyn Iout (t ) = Ig Iin I Ig Iin I e

0
t

Isyn =

UT

Isyn

e ( )d

Iw Igain I
(Bartolozzi, Indiveri, 2007)

Mean response to spike train of mean frequency : < Iout >=

G.Indiveri (NCS @ INI)

t , =

t + ISI
32 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

31 / 78

ICANN09 Tutorial

DPI measured response

DPI response to spike-trains

120 V =1.30V w V =1.33V w V =1.35V
w

300 Vw=420mV 250 200 150 100 50 0 0 Vw=440mV Vw=460mV EPSC (nA)

450 400 350 300 250 200 150 100 50 0.05 0.1 Time (s) 0.15 0 0 0.5 1 1.5 2 2.5 Time (s) 3 3.5 Vw=320mV Vw=340mV

Output Frequency (Hz)

Vw=300mV

100 80 60 40 20 0

EPSC (nA)

50

100 150 Input Frequency (Hz)

ICANN09 Tutorial

200

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

33 / 78

G.Indiveri (NCS @ INI)

34 / 78

Proc. Natl. Short-term depression Acad. Sci. USA 94 (1997)

Short-term depression
Vgain Va M6 M7 Id Vx M5 C Vpre Isyn M4 C2
Vx (V) 0.35
Slow recovery

l e e f e h

Va

Vw Cd
Vd

M1

0.3 0.25 Update 0.2

Vd=0.3 V Vd=0.26 V Vd=0.28 V Fast recovery

Ir

Vd

Ir

M2

0.15

e n

Vpre

M3

0.1 0.04

0.06

0.08

0.1 0.12 Time (s)

0.14

0.16

(a)

(b)

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

35 / 78

P,

p qYdi gh$

(C. Rasche, R. Hahnloser, 2001)

uS t

t S r sS @ S i Ygh$ f

Figure 1: Schematic for a depressing synapse circuit and responses to a regular input spike (M. Boegerhausen, P. Suter, and S.-C. Liu 2003) train. (a) Depressing synapse circuit. The voltage determines the synaptic conductance while the synaptic term or is exponential in the voltage, . The subcircuit consisting of transistors, , , and ICANN09 Tutorial the dynamics of , control . The presynaptic G.Indiveri (NCS @ INI) 36 / 78 input goes to the gate terminal of which acts like a switch. When there is a presynaptic

e"d

STDP and beyond

Alternative spike-driven learning algorithm
Spike-driven weight change depends on the value of the post-synaptic neurons membrane potential, and on its recent spiking activity.
Fusi et al. 2000; Brader et al. 2007

Spike-driven learning in VLSI I

J. Arthur and K. Boahen. Learning in silicon: Timing is everything. In Y. Weiss, B. Schlkopf, and J. Platt, editors, Advances in Neural Information Processing Systems 18. MIT Press, Cambridge, MA, 2006. A. Boll-i Petit and A. F. Murray. Synchrony detection and amplication by silicon neurons with STDP synapses. IEEE Transactions on Neural Networks, 15(5):12961304, September 2004. E. Chicca, D. Badoni, V. Dante, M. DAndreagiovanni, G. Salina, S. Fusi, and P. Del Giudice.

Recipe for efcient VLSI implementation

1 2

A VLSI recurrent network of integrateandre neurons connected by plastic synapses with long term memory. IEEE Transactions on Neural Networks, 14(5):12971307, September 2003. P. Higer, M. Mahowald, and L. Watts. A spike based learning neuron in analog VLSI. In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in neuralinformation processing systems, volume 9, pages 692698. MIT Press, 1997. G. Indiveri, E. Chicca, and R. Douglas. A VLSI array of low-power spiking neurons and bistable synapses with spiketiming dependent plasticity. IEEE Transactions on Neural Networks, 17(1):211221, Jan 2006. S. Mitra, G. Indiveri, and S. Fusi. Learning to classify complex patterns using a VLSI network of spiking neurons. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 10091016, Cambridge (MA), 2008. MIT Press.

bistability: use two synaptic states; redundancy: implement many synapses that see the same pre- and post-synaptic activity stochasticity & inhomogeneity: induce LTP/LTD only in a subset of stimulated synapses.

- Slow learning: only a fraction of the synapses memorize the pattern. + The theory is matched to the technology: use binary states, exploit mismatch and introduce fault tolerance by design.
G.Indiveri (NCS @ INI) ICANN09 Tutorial 37 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

38 / 78

Neurons . . . in a nutshell
A quick tutorial

Neurons of the world

Real Neurons Complexity Conductance based models Integrate and re models Rate based models
Sigmoidal units Linear threshold units

(adapted from B. Mel, 1994) G.Indiveri (NCS @ INI) ICANN09 Tutorial 40 / 78 G.Indiveri (NCS @ INI) ICANN09 Tutorial 42 / 78

Equivalent Circuit
Eex (Na+, ...) Glutammate

Spike generating mechanism

ENa gNa
Vmem Gl GABA Einh (K+, Cl, ...) Cmem

Vmem gK EK
If the membrane voltage increases above a certain threshold, a spike-generating mechanism is activated and an action potential is initiated.
G.Indiveri (NCS @ INI) ICANN09 Tutorial 44 / 78

Gl

Cmem

If excitatory input currents are relatively small, the neuron behaves exactly like a rst order low-pass lter.

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

43 / 78

Spike properties
Iin=I1

The F-I curve

Refractory Period

Pulse Width

Iin=I2 > I1

Spike Frequency (F)

Refractory Period

Input Current (I)

G.Indiveri (NCS @ INI) ICANN09 Tutorial 45 / 78 G.Indiveri (NCS @ INI) ICANN09 Tutorial 46 / 78

Hardware implementations of spiking neurons

Conductance-based models of spiking neurons

The rst articial neuron model was proposed in the 1943 by McCulloch and Pitts. Hardware implementations of this model date almost back to the same period.

Hardware implementations of spiking neurons are relatively new.

One of the most inuential circuits that implements an integrate and re (I&F) model of a neuron was the Axon-Hillock Circuit, proposed by Carver Mead in the late 1980s.

In 1991 Misha Mahowald and Rodney Douglas proposed a conductance-based silicon neuron and showed that it had properties remarkably similar to those of real cortical neurons.
G.Indiveri (NCS @ INI) ICANN09 Tutorial 49 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

47 / 78

Conductance based Si-Neurons

Vdd Sodium CurrentVdd

Conductance based Si-Neurons

Silicon neurons measurements

Sodium
1V Vm

+
VNa Vthr

+
GNaoff

INaoff

ENa

[Ca]

Vthr Vmem

INaon

INa

+
GNaon

Passive Leak
Vdd Eleak Vmem

Vm [Ca]

Vdd

+
Gleak Cmem

+
VK Vthr

Passive
IK GK EK EK IK

Vm [Ca]

Potassium

Potassium Current

I 50 ms
G.Indiveri (NCS @ INI) ICANN09 Tutorial 50 / 78 G.Indiveri (NCS @ INI) ICANN09 Tutorial 51 / 78

The Axon-Hillock Circuit

Positive Feedback

The Axon-Hillock Circuit

voltage Vout

Input current Membrane voltage

Vmem

Vout
Output voltage

Vmem

Vout

Vmem

time Vout
Reset

Vpw Slope = A

Vpw
Vmem

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

53 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

54 / 78

Capacitive Divider

Positive Feedback
Cfb

voltage

Vout

Given the change V 2, what is V 1? Q = C1 V1 + C2 (V1 V2 ) = constant C1 V1 + C2 (V1 V2 ) = 0

Q

C2

Vmem

Vout

Vmem

V1

A
C1 V2

Cm

time

V1 =

C2 C1 + C2

V2
Vpw

Positive Feedback Cfb Vmem = V Cm + Cfb dd

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

55 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

56 / 78

Axon-Hillock Circuit Dynamics

voltage Cfb Iin Vmem Vout

Gain
How to make voltage gain
Vmem

Vout

Cm tH Ir Vpw tL

time

A
Vdd

tL =

Cfb + Cm Iin

Vmem =

Cfb Iin

Vdd

tH =

Cfb + Cm Ir Iin

Vmem =

Cfb Ir Iin Iin

Frequency Iin
G.Indiveri (NCS @ INI)

Pulse width 1/Ir for Ir

ICANN09 Tutorial

Whats bad about this?

57 / 78 G.Indiveri (NCS @ INI) ICANN09 Tutorial 58 / 78

Power Dissipation
The Axon-Hillock circuit is very compact and allows for implementations of dense arrays of silicon neurons

Conductance-based models
Integrate and Fire vs Hodgkin-Huxley

Traditionally there have been two main classes of neuron models:

BUT
it has a major drawback: power consumption During the time when an inverter switches, a large amount of current ows from Vdd to Gnd.

Integrate and re (I-C)

Conductance-based (R-C)

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

59 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

61 / 78

Conductance-based models
Integrate and Fire vs Hodgkin-Huxley

An ultra low-power generalized I&F circuit

Adaptation PositiveFeedback
M5 Iin Vahp M11 M12 M14

But recently proposed models bridge the gap between the two:
Vrest

M6

M18 Ifb M15 M19 Vspk M16 Vrf M20

M1

DPI

Vthr_ahp Vmem Cmem

M7

M8

Vthr

M2

M3

Imem Iahp M22

Leak
Vtau M9 M4

Cahp M10

DPI
Vtau_ahp M13

M17

M21

RefractoryPeriod

(G. Indiveri, P. Livi, ISCAS 2009)

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

62 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

63 / 78

Generalized Integrate and Fire models can account for a very large set of DPI neuron sub-threshold equations behaviors captured by far more complicated Hodgkin-Huxley models.
Adaptation
M5 Iin Vahp

SPICE simulations

d dt
M6

umem =
M11 M12

PositiveFeedback in

M14

Cmem

+ F (umem )
20 Vthr=0.325 V Vthr=0.350 V Vthr=0.375 V
14 12 10

where F (umem ) is a non-linear function of umem (t ). DPI

Vrest M1 Vthr_ahp Vmem Cmem Imem Iahp M22 M7 M8 Ifb M15 Vthr M19 Vspk M16 Vrf M20 M2 M3

M18

15

Leak
Vtau M9 M4

Cahp M10

DPI
Vtau_ahp M13

Imem (A)

M21

Imem (A)

M17

10

8 6 4 2

RefractoryPeriod

0 2

d dt

Imem + Imem 1 I
1

Ig Iin I

+ (Imem )

6 Time (ms)

10

20

40 60 Time (ms)

80

100

+ I0 1 ,

2 + 1

+1

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

64 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

66 / 78

Experimental results
Single spike

Experimental results
Population activity
30 25 Neurons 20 15 10 5 30 25 20 15 10 5 0 0.5 1 1.5 2 0 0 0.5 1 1.5 2

2.8 2.7 2.6 Voltage (V) 2.5 2.4 2.3

Vthr= 0.0V V = 0.3V

thr

Vthr= 0.6V Vthr= 0.9V

30

30 25 20 15 10 5 0 0.5 1 Time (s) 1.5 2 0 0 0.5 1 Time (s) 1.5 2

70 / 78

2.2 2.1 2 1.9 0.01

G.Indiveri (NCS @ INI)

25 Neurons 20 15 10 5

0.02

0.03 Time (s)

ICANN09 Tutorial

0.04

0.05
67 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

Spiking multi-neuron architectures

Spikes and Address-Event Systems

1 0.8

Vmem (V)

0.6

0.4

Encode
0.2 0 0 0.05 0.1 Time (s) 0.15 0.2

Decode

1 2 3

Address Event Bus 1 23 2 3 12

1 2 3

Networks of I&F neurons with adaptation, refractory period, etc. Synpases with realistic temporal dynamics Winner-Take-All architectures Spike-based plasticity mechanisms

Inputs Source Chip

Outputs Destination Chip Address-Event representation of action potential Action Potential

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

72 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

73 / 78

Spikes per seco

20 15 10 5 0 180 120 60 0 60 120

239

A spike-based learning chip

180

Rotation around y axis (degrees)

A minimum-size chip implementing a recongurable AER neural network. Neurons and synapses have realistic temporal dynamics. Local circuits at The basic each synapse implement theproblem with these models is, of course, generalization: bi-stable a look-up table cannot deal with new events, such as viewing a face from the side rather than the front, and it cannot learn in the predicspike-based plasticity mechanism. One of the simplest and most powerful tive sense described earlier.
Indiveri, Fusi, 2007
types of algorithm developed within learning theory corresponds to networks that combine the activities of units, each broadly tuned to one of the examples (Box 1). Theory (see references in Box 1) shows that a combination of broadly tuned neurons those that respond to a variety of stimuli, although at sub-maximal firing rates might generalize well by interpolating among the examples. In visual cortex, neurons with a bell-shaped tuning are common. Circuits in infratemporal cortex and prefrontal cortex, which combine activities of neurons in infratemporal cortex tuned to different objects (and object parts) with weights learned from experience, may underlie several recognition tasks, including identification and categorization. Computer models have shown the plausibility of this scheme for visual recognition and its quantitative consistency with many data from physiology and psychophysics25 . Figure 2 sketches one such quantitative model, and summarizes a set of basic facts about cortical mechanisms of recognition established over the last decade by several physiological studies of cortex68. Object recognition in cortex is thought to be mediated by the ventral visual pathway running from primary visual cortex, V1, over extrastriate visual areas V2 and V4 to the inferotemporal cortex. Starting from simple cells in V1, with small receptive fields that respond preferably to oriented bars, neurons along the ventral stream show an increase in receptive field size as well as in74 /complexity of their preferred stimuli. the 78 At the top of the ventral stream, in the anterior inferotemporal cortex, neurons respond optimally to complex stimuli such as faces and other objects. The tuning of the neurons in anterior inferotemporal cortex probably depends on visual experience919. In addition, some neurons show specificity for a certain object view or lighting condition13,18,2022. For example, Logothetis et al.13 trained monkeys to perform an object recognition task with isolated views of novel three-dimensional objects (paperclips; Fig. 1). When recording from the animals' inferotemporal cortex, they found that the great majority of neurons selectively tuned to the training objects were view-tuned (see Fig. 1) to one of the training objects. About one tenth of the tuned neurons were view-invariant, consistent with an earlier computational hypothesis23.
NATURE | VOL 431 | 14 OCTOBER 2004 | www.nature.com/nature

Figure 1 Tuned units in inferotemporal cortex. A monkey was trained to recognize a three-dimensional paperclip from all viewpoints (pictured at top). The graph shows tuning to the multiple parameters characterizing each view summarized in terms of spike rate versus rotation angle of three neurons in anterior inferotemporal cortex that are view-tuned for the specific paperclip. (The unit corresponding to the green tuning curve has two peaks to a view of the object and its mirror view.) A combination of such view-tuned neurons (Fig. 2) can provide view-invariant, object specific tuning as found in a small fraction of the recorded neurons. Adapted from Logothetis et al.13.

which several models have been suggested24; see also review in this issue by Abbott and Regehr, page 796), since it depends only on passive experience of the visual inputs. However, the weights of the combination (see Fig. 3) depend on learning the task and require at least some feedback (see Box 2). Thus, generalization in the brain can emerge from the linear combination of neurons tuned to an optimal stimulus effectively defined by multiple dimensions25,23,26. This is a powerful extension of Analog processing, asynchronous digital the older computation-through-memory models of vision and motor control. The question now is whether the available evidence supports the existence of a similar architecture underlying generalization in domains other than vision.

Distributed multi-layer networks

communication

AER INPUT Y

PFC

Categ.

Ident.

AER OUTPUT

AIT

IT

V4/PIT

Technology: Size: Neurons: AER plastic synapses: AER non-plastic synapses Dendritic tree multiplexer:

AMS 0.35 m 3.9mm 2.5mm 128 28 128 4 128 32 128 | . . . | 1 4096

V4

V1

V1

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

Summary
Vg Vs Ir Vs Ir
Qs

Thank you for your attention

Additional information available at:

Figure 2 A model of visual learning. The model summarizes in quantitative terms other models and many data about visual recognition in the ventral stream pathway in cortex. The correspondence between the layers in the model and visual areas is an oversimplification. Circles represent neurons and arrows represent connections between them;G.Indiveri (NCS @ INI) same type. Stages of neurons the dots signify other neurons of the with bell-shaped tuning (with black arrow inputs), that provide example-based learning and generalization, are interleaved with stages that perform a max-like operation3 (denoted by red dashed arrows), which provides invariance to position and scale. An experimental example of the tuning postulated for the cells in the layer labelled inferotemporal in the model is shown in Fig. 1. The model accounts well for the quantitative data measured in view-tuned inferotemporal cortex cells10 (J. Pauls, personal communication) and for other experiments55. Superposition of gaussian-like units provides generalization to three-dimensional rotations and together with the soft-max stages some invariance to scale and position. IT, infratemporal cortex, AIT, anterior IT; PIT, posterior IT; PFC, prefrontal cortex. Adapted from M. Riesenhuber, personal communication.

ICANN09 Tutorial

AER INPUT X

75 / 78

Vd If Vg Vd If
Qd

2004 Nature Publishing Group

http://ncs.ethz.ch/

769

Adaptation
M5 Iin Vahp M11 M12

PositiveFeedback
M14

M6

M18 Ifb M15 M19 Vspk M16 Vrf M20

Vrest

M1

DPI

Vthr_ahp Vmem Cmem

M7

M8

Vthr

M2

M3

Imem Iahp M22

Leak
Vtau M9 M4

Cahp M10

DPI
Vtau_ahp M13

M17

M21

RefractoryPeriod

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

77 / 78

G.Indiveri (NCS @ INI)

ICANN09 Tutorial

78 / 78