Vous êtes sur la page 1sur 102

Long Short-Term Memory in Recurrent Neural Networks

`

TH ESE N 2366 (2001)

´

´

´

PR ESENT EE AU D EPARTEMENT D’INFORMATIQUE

´

´

´

ECOLE POLYTECHNIQUE F ED ERALE DE LAUSANNE

`

POUR L’OBTENTION DU GRADE DE DOCTEUR ES SCIENCES

PAR

FELIX GERS

Diplom in Physik, Universitat¨ Hannover, Deutschland de nationalite´ allemand

soumise a` l’approbation du jury:

Prof. R. Hersch, president´

Prof. Wulfram Gerstner, directeur de these`

Dr. habil. Jurgen¨

Schmidhuber, corapporteur

Prof. Paolo Frasconi, corapporteur Dr. MER Martin Rajman, corapporteur

Lausanne, EPFL

2001

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Out Out In In
Out
Out
In
In
Out Out In In
Out
Out
In
In

Out Out In In Out Out In In

Out In Output Hidden Input Out Memory Output Gate Block with one Cell InputGate In

Out In
Out
In

Output

Hidden

Out In Output Hidden Input Out Memory Output Gate Block with one Cell InputGate In

Input

Out Memory Output Gate Block with one Cell InputGate In
Out
Memory
Output Gate
Block
with
one
Cell
InputGate
In

y c y out y out output gating h w net out out ouput gate
y
c
y
out
y out
output gating
h
w
net
out
out
ouput gate
h(
s
)
output squashing
c
s = s + g y
in
1.0
c
c
CEC: memorizing
y
in
g
y in
input gating
w
net
in
in
input gate
g(net )
input squashing
c
w
c
net
c

y
y

c

y c y out h y out output gating net w out ouput gate h( c
y out h y out output gating net w out ouput gate h( c )
y
out
h y out
output gating
net
w out
ouput gate
h( c )
s
output squashing
ϕ
y
s = s y + g y
ϕ
in
c
c
w
net
ϕ
memorizing and forgetting
forget gate
y
in
g y in
input gating
net
w in
input gate
g(net )
input squashing
c
w c net c
w
c
net
c

out

ϕ

in