Vous êtes sur la page 1sur 15

TOWARD MULTI PARADIGM COMPUTING

Symposium on Emerging Trends in Computing, Montreux, Switzerland | Christian GAMRAT | October 2016

MY BRIEF HISTORY OF COMPUTING.

1978 - 1985
Microprocessors
1978
PDP-11

1983 - 1989
Multiprocessors

Analog

Digital
1985 - 1994
Neural Networks

1988
Cellular Automata
|2

MY BRIEF HISTORY OF COMPUTING, CONTD.

1995 - 1998
SIMD for Image processing

1998 - 2005
Reconfigurable Computing

2002
A bit of Quantum

-6

10

(a)

W21

(b)

-7

10

Bias-
Function output

ID(A)

X1+

X1- X2+ X2-


X3+ X3- Bias+

-8

10

-9

10

VG
-3

-2

-1

VGS (V)

2003 -> Now


Neural Nets are back
With Nanotech and DNN

2006 LSM
a.k.a. Reservoir Computing

2016 guess what?


Quantum is back!....
|3

WHY DO WE DO ALL THOSE CRAZY THING?

Because its fun.


Because we seek for the ultimate way to process information?
Because we are not satisfied with the computers we have?

Because computers are stupid (are they really?)

They do reproduce faithfully the mistakes of their programmers

Because they consume a lot of energy flipping too many bits?


Because they cannot handle certain tasks?
Because they separate memory and operators?
Because they need to be programmed?

|4

A CLOSER LOOK AT THE KING OF THE PARADIGMS


Why did they
name this
after me?

Physical
Implementation
(HW)
John Von Neumann

Implementation
Technology

Information
Coding

Computing
Paradigm

Computing
Architecture

Programming
Model (SW)

0 1
|5

A WORD ABOUT INFORMATION CODING

We can handle various ways of representing information

Fusing coding schemes often means transcoding

Digital, analog, streams, bit-serial

But it is not always possible


For quantum state we prepare initial Q states and readout classical states

Abstracting data from the physical implementation

Encode in the correlation of several physical entities


Encode in a probability, stochastic coding

0 1

0 1
Causality

Anti-causality
|6

FUSING PARADIGMS @HARDWARE LEVEL


At the hardware level, the good old V.N./ CMOS partnership
can act as a computing substrate
Acting as coordination / communication node
Allowing Hardware / Software integration
Neuro
Engine

Qubits on Silicon

Quantum
Engine

Coordination Engine
Von Neumann style
CMOS Technology
binary data

M. Veldhorst et al,
Nature, 526, Oct. 2015.

Graphical
Engine

NVM Synapses
on Silicon

Decision
Engine

Semantical Numerical
Engine
Engine

Numerical
Neuro
Quantum
Graphics
Engine
engine
engine
engine
Physical and logical interface layer
CMOS Substrate

D. Roclin et al, IEEE


NanoArch, 2014.
|7

COORDINATE PARADIGMS @SOFTWARE LEVEL

A sequential program (running on the


coordinator) distributes tasks to engines

Tasks are distributed to computing engines

Accelerator as services (J. Cong)


Instruction streams
Supervised Learning
Unsupervised Learning
Quantum Engine

Why would we want that?

Express
Run hard tasks, speedup NP problems
Provide cognitive functions

Stefanini et al. Frontiers Neuroinformatics


August 2014

Road blocks!..

Valiron et al., Programming the Quantum Future,


Commun. ACM, vol. 58, no. 8, pp. 5261, Jul. 2015.

Software communities are very


conservative
We tend to think sequentially
Security / trust concerns

Fu et al. Proc. ACM, May 2016

|8

MULTI PARADIGM COMPUTING

Emerging computer: Is it programmable? Can it learn? Can we use it?


Most emerging computing paradigm need one classical host (at least)

Todays hybrid computing efforts

Programing homegeneous parallel computers efficiently is just becoming


possible
We still struggle on the efficient programming of heterogeneous computers
(CPU, GPU, FPGA, DSP)
We are scratching the surface of how to program or teach neuromorphic
circuits using supervised algorithms
So How will me mix sequential programs with unsupervised learning?
And what about introducing quantum accelerators..

The coordination of multiple paradigms, multiple coding schemes,


multiple technologies will be key. Its a system & software issue
A specific action in multi paradigms computing is needed
|9

Leti, technology research institute


Commissariat lnergie atomique et aux nergies alternatives
Minatec Campus | 17 rue des Martyrs | 38054 Grenoble Cedex | France
www.leti.fr

LA DICHOTOMIE APPRENTISSAGE / RELAXATION

Borrowed from Yann le Cun, 2016, FAIR labs


| 11

UNSUPERVISED STDP LEARNING


Unsupervised learning paradigm exploration
"
"

Spike-Timing-Dependent Plasticity is a fundamental learning rule in the brain


Unsupervised extraction of correlations in sequences

Neuron
Electrical
signal
Axon
Dendrite

"

Synapse

tpost

Pre-synap7c Post-synap7c
neuron
neuron

Application example
"
"

CMOS
retina

<

tpre

Synaptic weight modification


(%)

"

Synaptic activation
correlated with
neuron activation
tpre

<

tpost

Unsupervised learning of traffic lanes using asynchronous spiking CMOS retina


t = tpost - tpre
Network of 70 Leaky-Integrate and Fire (LIF) neurons and ~2M synapses

Lateral
inhibi7on

2nd layer
1st layer


CMOS Re(na
16 384 pixels

128

Traffic lanes
visualization
Neuron sensitivity
maps
[O. Bichler et al., Neural Networks, 2012]

| 12

SPIKE-CODING FOR DEEP NETWORKS


The promises of spike-coding NN:
"
"

Reduced computing complexity and natural temporal and spatial parallelism


Simple and efficient performance tunability capabilities
Spiking NN best exploits NVMs such as RRAM, for massively parallel synaptic memory
Base operation
Activation
function
Parallelism

1) Standard CNN topology, offline


learning
Conv. layer
16 maps
11x11
Input: digit
neurons
24x24 pixels

Conv. layer
24 maps
4x4
neurons

(cropped)

16 kernels
4x4 synapses

Formal neurons
- Multiply-Accumulate
(MAC)
- Non-linear function

Spiking neurons
+ Accumulate only

- Spatial multiplexing

+ Spatial and temporal


multiplexing

+ Simple threshold

2) Lossless spike
transcoding
Pixel
brightness

Spiking frequency
V
fMIN

Rate-based
input
coding

layer 1

layer 2

fMAX

layer 3

layer 4
Correct
Output

3) Performance vs computing time


tunability (approximated
computing)
5

2.5

4.5
4

3.5
3

1.5

2.5
2

1.5
1

0.5

0.5
0

0
1

90 kernels
5x5 synapses

Spikes / connection

"

Test error rate (%)

"

9 10

Decision threshold
Time

[O. Bichler et al., IEDM, 2015]

| 13

NVM SYNAPSES IMPLEMENTATIONS


"

2-PCM synapses for unsupervised cars trajectories extraction


From spiking pre-synap7c
neurons (inputs)
VRD
ILTP

PCM

ILTD
I = ILTP - ILTD
Spiking post-
synap7c neuron
Equivalent
(output)
2-PCM synapse

Crystallization/
Amorphization

[O. Bichler et al., Electron Devices, IEEE TransacAons on, 2012]

"

CBRAM binary synapses for unsupervised MNIST handwritten digits


classification with stochastic learning
CBRAM

Forming/Dissolution of
conductive filament

[M. Suri et al., IEDM, 2012]


| 14

PLAYING WITH UNSUPERVISED LEARNING


Lateral
inhibition

2nd layer
1st layer

Lateral
inhibition

AER Sensor

128

16,384 spiking pixels

128
Two-layers system
~ 2 million devices with STDP

O. Bichler, D. Querlioz, S. J. Thorpe, J.-P. Bourgoin and C.


Gamrat, Unsupervised Features Extraction from
Asynchronous Silicon Retina through Spike-Timing-Dependent
Plasticity, International Joint Conference on Neural Networks
IJCNN August 2011
| 15