14 D

Dynamic Networks

1
14 Dynamic Networks D

Dynamic networks are networks that contain

delays and that operate on a sequence of inputs.
The ordering of the inputs is important to the
operation of the network.
In dynamic networks, the output depends not only
on the current input to the network, but also on the
current or previous inputs, outputs or states of the
network.

2
14 Layered Digital Dynamic Networks D

Inputs Layer 1 Layer 2 Layer 3

T
1,1
D LW
L a1(t)
1
p (t)
1 n (t) 2
a (t)
3
a (t)
T
IW1,1 D LW
2,1
2 LW
3,2
n3(t)
n (t)
1
R x1 S2x1
1 L 2 3
3
S x1
1
S xR f 1
S x1
2
S xS 1

2
f 3
S xS 2

3
f
S x1 S x1
1 b1 S1x1 1 b
2
1 b
3

1 2 3
R S1x1 S1 S2x1 S S3x1 S
T T
D LW1,3 D LW2,3
L L

n m t = L Wm l d al t d + IW m l d p l t d + b m

l Lm d D Lm l l I m d D Im l
f

a m t = f m n m t
3
14 Layer Components D

A set of weight matrices that come into that layer

(which may connect from other layers or from
external inputs),
Any tapped delay lines that appear at the input of a
set of weight matrices,
A bias vector,
A summing junction, and
A transfer function.

4
14 Definitions D

Simulation Order
Backpropagation Order
Input Layer (has an input weight, or contains any
delays with any of its weight matrices)
Output Layer (its output will be compared to a
target during training, or it is connected to an input
layer through a matrix that has any delays
associated with it)

5
 14 Example Feedforward Dynamic Network D

Inputs Linear Neuron 1.5

p(t) iw1,1(0)
0.5

D iw1,1(1) n(t) a(t) 0

S 0.5

D 1

iw1,1(2)
1.5
0 5 10 15 20 25

a(t) = iw1,1(0)p(t)+iw1,1(1)p(t-1)+iw1,1(2)p(t-2)

1 1 1
iw 1 1 0 = --- iw 1 1 1 = --- iw 1 1 2 = ---
3 3 3

6
14 Example Recurrent Dynamic Network D

Inputs Linear Neuron 1.5

iw1,1 n(t) a(t)

p(t)
S 0.5

lw1,1(1) 0

D 0.5

a(t) = iw1,1 p(t) + lw1,1(1) a(t-1) 1.5

0 5 10 15 20 25

1 1
lw 1 1 1 = --- iw1 1 = ---
2 2

7
14 Applications of Dynamic Networks D

Prediction in financial markets

Channel equalization in communication systems
Phase detection in power systems
Nonlinear filtering/fusion of sensor signals
Fault detection
Speech recognition
Prediction of protein structure in genetics

8
14 Focused Time Delay Network D

Inputs Layer 1 Layer 2

p1(t)
T a1(t) a2(t)
D IW
1,1
n1(t) LW
2,1
2
L 1
1
S x1 n (t) 2
S2x1

d
S 1 x (R1d) f S2xS1
2
f
S x1
1 b1 1
S x1 1 b
2

R1 S1x1 S1 S2
2
S x1

9
14 Distributed Time Delay Network D

Inputs Layer 1 Layer 2

1
p (t) 1
a (t) 2
T T a (t)
1,1
D IW 1
D LW
2,1
2
n (t) n (t)
2
S x1
L 1 L 2
1
S 1 x (R1d 1) f 2
S 2 x (S 1d 2) f
d d S2x1
1 b
1
S1x1 1 b2
1 2
R1
1
S x1 S S2x1 S

10
14 Layered Recurrent Network D

(Elman)

Inputs Layer 1 Layer 2

D LW1,1
1 1
p (t)1 n (t) a (t) 2
a (t)
1,1
IW LW
2,1
2
n (t)
1 2
R x1 S x1
1 2
f f
1
S1xR S x1 2
S xS
1

2
S x1
1 b
1
S1x1 1 b 2

1 1 2 2
R S1x1 S S x1 S

11
14 LRN Example D

Network Input Network Response

1.5 1.5

1 1

0.5 0.5

0 0

0.5 0.5

1 1

1.5 1.5
0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400

12
14 Nonlinear ARX D

y t = f y t 1 y t 2 y t n y u t 1 u t 2 u t nu

Inputs Layer 1 Layer 2

p1(t) = u(t) n1(t) a1(t) a2(t) = ^y(t)

T
D IW1,1 LW
2,1
2
L 1 S1x1 n (t) 2 S2x1
1
R x1
1
S xR f 2
S xS
1
f
S2x1
1 b1 S1x1 1 b2
1 1
S2
2
R S1x1 S S x1
T
D LW1,3
L

13
14 Modeling with NARX D

N 6

Position
4

S 2

0
0 500 1000 1500 2000 2500 3000 3500 4000

yt 4

i(t) 3

+ 2

Voltage
1

- 1
0

0 500 1000 1500 2000 2500 3000 3500 4000

0.5

0.01

0.005
0

0.5
0.005

0.01

1
0 500 1000 1500 2000 2500 3000 3500 4000 0 100 200 300 400 500 600 700 800 900

14
 14 Model Reference Control D

T ec(t)
D LW1,2 a2 (t)
L
r(t) n1(t)
T c(t)
D IW 1,1 LW2,1 n2(t) Plant
L
f1 f2 ep(t)
1 b1 1 b2

a3 (t) y(t)
T T
D LW1,4 D LW3,2 n3(t) LW4,3 n4(t)
L L
f4 a (t)
4
ff32

1 b3 1 b4

T
D LW3,4
L

Neural Network Controller Neural Network Plant Model

15
 14 Example of Dynamic Derivatives D

Inputs Linear Neuron

Q Q
2 2
iw1,1 n(t) a(t) F x = e t = t t a t
p(t)
S t= 1 t= 1
lw1,1(1)
D Q
2
Q
F x----
------------ ------------t--- = 2
-e ----------t----
--a
iw1 1
= iw1 1 iw1 1
e t
t=1 t=1
a(t) = iw1,1 p(t) + lw1,1(1) a(t-1)

Q Q
2
---------x-------- =
F
------- ----------t------- = 2
e
------- --------t--------
a
--------
l w1 1 1 lw 1 1 1 lw 1 1 1
e t
t= 1 t =1

--------t-------- = a t 1 + lw 1 --a
a
-------- --t----
-------- 1----
---- ---------t----
--a - a t 1
= p t + lw1 1 1 ----------------------
lw 1 1 1 1 1 lw 1 1 1 iw 1 1 iw1 1

16
14 iw1,1 Effect D

2 1.5

Total Derivative
1.5
Static Derivative
1

0.5 Original Response

0.5 Incremental Response

0 0

0.5
0.5

1
1.5

2 1.5
0 5 10 15 20 25 0 5 10 15 20 25

17
14 lw1,1(1) Effect D

2
1.5

1.5

Original Response
0.5
0.5 Incremental Response

0
0

0.5

0.5

1
1.5 Total Derivative
Static Derivative

2
0 5 10 15 20 25 1.5
0 5 10 15 20 25

18
14 Basic Concepts of Dynamic Training D

Simple Recurrent Network

p(t)
a(t)
Multilayer Network

a(t) = NN(p(t),a(t-1),x)
D

Performance Index
Q
T
F x = t t a t t t a t
t =1

19
14 Dynamic Backpropagation D

Real Time Recurrent Learning (RTRL)

Q
e
----F
-- = --a------t---
--
T F
-------------
x x
T a t
t =1

e e
---a
--------t- = -------a------t-- + ------------a------t------- -----
a------t--------1---
T T T T
x x a t 1 x

Backpropagation Through Time (BPTT)

Q T
e
F-
----- ------a------t--- F
= -------------
x x
T a t
t= 1

e
---- + -------a------t----
e
-------F------ = ---------F +----1---- ------------F
-----------

a t a t a t
T a t + 1
20
14 General RTRL D

Q Q u T
e e
----F ----a------t--- T F -----F- = ----a--------t--- F
-- =
------------- ---------------
x x
T a t x x
T u
a t
t =1 t = 1u U

e e
---a
--------t- = -------a------t-- + ------------a------t------- -----
a------t--------1---
T T T T
x x a t 1 x

u e u e u e x u'
----a----------
t -- -------a--------t--- a---------t - -------------n--------t------- ----a----------t--------d---
--------
= +
T T x T T T
x x u' U x X d DL x u' n t a u'
t d x

21
14 Sensitivities D

e u e x
a---------t - -----------n
-------- ----------t------- = S u x t L Wx u' d

x T u' T
n t a t d

u m u m u m
s 1 1 t s 1 2 t s 1 S m t

a t u m u m u m
e u
u m
S s
t = ------------------T- = 2 1 t s 2 2 t s 2 S m t
n t
m

u m u m u m
sS u 1 t sS u 2 t s S u Sm t

m u
S u m t = S u l t LW l m 0 F n m t S u u t = F n u t
b
l E S u Lm
22
14 Explicit Derivatives D

e u e u e m
a t ak t n t
------- ------------i------------ = su m t p l t d
-------------k----------- = -----------
m l m m l k i j
iw i j d ni t iw i j d

e u
--------------------a--------t----
------------ l T u m
= p t d S t
m l T
ve c IW d

e u
--------------------a
----------t---------------- = al t d Su m t
T
m l T
v ec L W d
e u
------a
----------t-- = Su m t
m T
b

23
14 RTRL Example (1) D

Inputs Linear Neuron

iw1,1 n(t) a(t)

p(t)
S
lw1,1(1)
D

a(t) = iw1,1 p(t) + lw1,1(1) a(t-1)

Q 3
2 2 2 2 2
F = t t a t = e t = e 1 + e 2 + e 3
t =1 t= 1

p 1 t 1 p 2 t 2 p 3 t 3

a 1 = lw1 1 1 a 0 + iw1 1 p 1
24
14 RTRL Example (2) D

S 1 1 1 = F n1 1 = 1
1

e 1 e
a--------1---------------- -------a------1---
------ T
-------------- = = p 1 1 S1 1 1 = p 1
I W 1 1 T iw1 1
ve c 0
e 1 e
a--------1----------------- -----------a------1-------
--------------------- 1 T
S1 1 1 = a 0
= = a 0
1 1 T lw1 1 1
v ec LW 1

1 e 1 1
----a--------t--- a t 1 1 1 1 a t 1
= ------------------ + S t LW 1 -------------------------
T T T
x x x

----------1----- = p 1 + lw 1 --a
-a --------0------ = p 1
iw 1 1 1 1 iw 1 1

--------1---------- = a 0 + lw 1 ------a
------a --------0---------- = a 0
lw1 1 1 1 1 lw1 1 1
25
14 RTRL Example (3) D

e e
-------a------2--- = p 2 ----------a------2-------- = a 1
iw1 1 lw1 1 1

--------2------ = p 2 + lw 1 --a
--a ----------1--- = p 2 + lw 1 p 1
iw 1 1 1 1 iw1 1 1 1

--------2--------- = a 1 + lw 1 -------a
-------a --------1--------- = a 1 + lw 1 a 0
lw1 1 1 1 1 lw1 1 1 1 1

e e
-------a------3--- = p 3 ----------a------3-------- = a 2
iw1 1 lw1 1 1

----------3----- = p 3 + l w 1 --a
-a --------2------ = p 3 + lw 1 p 2 + lw 1 2 p 1
iw 1 1 1 1 iw1 1 1 1 1 1

-------a 3-------
---------- a 2 2
= a 2 + lw 1 1 1 ------------------------ = a 2 + lw 1 1 1 a 1 + lw1 1 1 a 0
lw1 1 1 lw1 1 1
26
14 RTRL Example (4) D

Q u T 3 1 T
e e
F-
----- ----a--------
t -
-- F -----a--------t--- F
= --------------- = ---------------
x x
T
a t
u
x
T
a t
1
t = 1u U t=1

--------1------ 2e 1 + -a
------- = --a
---------F ---------2------ 2e 2 + ---
--------3----- 2e 3
a
iw 1 1 iw1 1 iw 1 1 iw 1 1
= 2e 1 p 1 2e 2 p 2 + lw1 1 1 p 1
2
2e 3 p 3 + lw 1 1 1 p 2 + lw1 1 1 p 1

--------1---------- 2e 1 + -------
------------F------------ = ------a --------2--------- 2e 2 + ------a
a --------3---------- 2e 3
lw1 1 1 lw1 1 1 lw1 1 1 lw1 1 1
= 2e 1 a 0 2e 2 a 1 + lw1 1 1 a 0
2
2e 3 a 2 + lw 1 1 1 a 1 + lw1 1 1 a 0

27
14 General BPTT D
Q T
e
F-
----- ------a------t--- F
= -------------
x x
T a t
t= 1

Q Su
e u e m
a t n t
-------------F
----------- = ---------F------ ---------k--------- ------------i------------
m l
lwi j d
u m
ak t ni t lwi j d
m l
t = 1 u Uk = 1

e
-------F------ = ---------F---- + -------a------t----
e
+----1---- ------------F
-----------

a t a t a t
T a t + 1

e e u' e x T
--------F
------- = -----------F---- + -------a-------- ------d--- ------n
--t--+ t --+
---------- ------d---- ----F
--------- -------------
a t
u
a t
u n x
t + d
T
a
t
u T u'
a t + d
u' U x X d D Lx u

e u' e x
-------a-------- +----d---- -------n--------t----+------
--t----

d---
= S u' x t + d LWx u d
x T u T
n t + d a t
28
14 BPTT D

Q Su
e u e m
a t n t
-------------F
----------- = ---------F------ ---------k--------- ------------i------------
m l
lwi j d
u m
ak t ni t lwi j d
m l
t = 1 u Uk = 1

dm t = u m T F
S t ---------------
u
u U a t

Q
F------------
----------------- d m t al t d
T

LW d
m l
=
t=1
Q
---------------F------------- =
T
d m t p l t d
I W d
m l
t= 1
Q
F---
------- d m t
b
m
=
t=1
29
14 BPTT Example (1) D

a 1 = lw1 1 1 a 0 + iw1 1 p 1

a 2 = lw1 1 1 a 1 + iw1 1 p 2

a 3 = lw1 1 1 a 2 + iw1 1 p 3

S 1 1 3 = F n1 3 = 1
1

e
--F
------ ------- = ----------F----- + LW1 1 1 S 1 1 t + 1 ------------F
T T
-------------
1 1 1
a t a t a t + 1
0
e eF
---------F------- = -----------F----- + lw 1 S 1 1 4 T ----------F------- = ------
------
----- = 2e 3
1 1 1 1 1 1
a 3 a 3 a 4 a 3

d1 3 = S1 1 3 --------F
T
-------- = 2e 3
1
a 3
30
14 BPTT Example (2) D

S 1 1 2 = F n1 2 = 1
1

e
---------F------- = -----------F----- + lw 1 S 1 1 3 ----------F-------
T
1 1 1 1 1
a
2 a
2 a 3
= 2e 2 + lw 1 1 1 2e 3

F------
d1 2 = S1 1 2 ----------
T
= 2e 2 + lw 1 1 1 2e 3
1
a 2

S 1 1 1 = F n1 1 = 1
1

e
---------F-------- = -----------F------ + lw 1 S 1 1 2 ----------
F-------
T
1 1 1 1 1
a 1 a 1 a 2
2
= 2e 1 + lw1 1 1 2e 2 + lw1 1 1 2e 3

d1 1 = S1 1 1 --------F
T
-------- = 2e 1 + lw 1 2e 2 + lw 1 2 2e 3
1 1 1 1 1
a 1
31
14 BPTT Example (3) D

3
F-------------
---------------- F d1 t a 1 t 1
T
= ------------------------ =
LW 1
1 1 lw 1 1 1
t=1
2
= a 0 2e 1 + lw 1 1 1 2e 2 + lw 1 1 1 2e 3
+ a 1 2e 2 + lw1 1 1 2e 3 + a 0 2e 3

3
--F-------------
------------- F d 1 t p 1 t
T
= --------------- =
I W 0
1 1 iw1 1
t= 1
2
= p 1 2e 1 + lw1 1 1 2e 2 + lw1 1 1 2e 3
+ p 2 2e 2 + lw 1 1 1 2e 3 + p 3 2e 3

32
14 D

A Problem with Recurrent Network

Training

Spurious Valleys

33
14 Recurrent Net Error Surface Profile D

4.2

4.1

3.9
mse

3.8

3.7

3.6

3.5

3.4
0 0.2 0.4 0.6 0.8 1
12
x 10
34
14 Simple Recurrent Network D

Inputs Linear Neuron

iw1,1 n(t) a(t)

p(t)
S
lw1,1(1)
D

a(t) = iw1,1 p(t) + lw1,1(1) a(t-1)

35
14 Mean Square Error Surface D

Training data generated with weight values:

lw = 0.5 iw = 0.5

lw

iw
lw
iw
Spurious Valleys

36
 14 Network Response D

a t iw1,1 p t lw1,1 1a t 1

2 t 1

a t iw p t lw p t 1 lw p t 2 lw p 1 lw a 0
t

The response can be considered a polynomial in lw.

The coefficients of the polynomial involve the input
sequence and the initial conditions.
The network output will be zero at the root of the
polynomial.
Roots greater than 1 (unstable system) tend to remain
constant with sequence length (increasing order).

37
14 Effect of Sequence Length D

Root of Polynomial

38
14 Cause of Spurious Valleys D

Initial Conditions
If some initial conditions (neuron outputs) are zero,
then there are certain combinations of weights that will
produce zero outputs for all time.
Input Sequence
There are values for the weights that produce an
unstable network, but for which the output remains
small for a particular input sequence. If the input
sequence is modified, it may produce a valley in a
different location.
39
14 Nonlinear Recurrent Network D

Inputs Tan-Sigmoid Neuron

iw1,1 n(t) a(t)

p(t)
S
lw1,1(1)
D Training Data (Skyline)
1

0.8

a(t) = tansig( iw1,1 p(t) + lw1,1(1) a(t-1) ) 0.6

0.4

0.2

0.2

0.4

0.6

0.8

1
0 2 4 6 8 10 12 14 16 18 20

40
14 Steepest Descent Trajectory D

1.5

0.5

lw 0

0.5

1.5
1.4

1.2
2
2 1.5 1 0.5 0 0.5 1 1.5 2
1

iw 0.8

0.6

0.4

0.2

0
2

1 2
1
0
0

lw 1
1
2 2
iw
41
14 Nonlinear Error Surface D

42
14 Different Input Sequence D

43
14 Changing Input Sequence D

44
14 Changing Initial Condition D

45
14 Procedures for Training Recurrent Nets D

Switch training sequences often during training.

Use small random initial conditions for neuron
outputs and change periodically during training.
Use a regularized performance index to force
weights into stable region. Decay regularization
factor during training.

J w = SSE + SSW

46
14 Summary D

Recurrent networks can be used for a variety of

filtering and control applications.
The gradient calculations for recurrent networks
require dynamic backpropagation.
The error surfaces of recurrent networks have
spurious valleys, which require modified training
procedures.

47