Sebastien
Marcel
marcel@idiap.ch
IDIAP Research Institute
Martigny, Switzerland

http://www.idiap.ch

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.1/47


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.2/47

!



 




 












 

 



 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.3/47

!



 




 






 








 



 


















 

 



 



 


 


generalize

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.4/47



 !










 

 









 



 















!







 

 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.5/47

&

' 
(

%$

" !
#







 























 








 











Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.6/47








 










 








!

!











 



 



!















 


 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.7/47






!





 









 









 





 



 



!



 




 




Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.8/47

!



 


 











 












 




















 












 

 



 













































Core









Axons
terminaisons

Soma
Axon

Dendrite

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.9/47







 









Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.10/47






 




 





















 















 



 












 



















 





 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.11/47



 


 




















 









 










 



 






 


 



 






 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.12/47























 


















 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.13/47

 


 





 

 










Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.14/47



!





 





















 






 














Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.15/47

 















 




Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.16/47

 






















!





!





 





Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.17/47

 



!





 



 
















'




 !






'






 

















 









Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.18/47

!



 


 











 












 




















 












 

 



 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.19/47
















 




"



"






!



 



!



 

 

 




 

 




 


 







 







 

 






 


 




















 













 


 


 













Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.20/47








 






 






 


 



 




!



 











!






 

 




 



 











Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.21/47

!



 





















 






 


 

 




Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.22/47


! 

"
#

"
"$
%


"












 










 





























 










 


 







!

(1)

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.23/47








(2)

(3)

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.24/47











Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.25/47







 





 












!






(4)

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.26/47

 


 


 




 







!

!











!

 



 





(5)

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.27/47






 










!







(6)

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.28/47



























 


 




 







 

 



(7)

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.29/47












Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.30/47



 

















 
















 








 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.31/47

 




















































 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.32/47

 

 

























 






























Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.33/47
W





 2

OB

HR

HR

GF

SG

J
OP
?
B



8: 7
9

5 3
6

= <
>

D
J

B H

GF
?

43

K ?
L

@B
A
?

 2

1

 0



'

!
"


 (

 





&

'

'




!
) "

!
) "

&

&

'

'

'




!
) "

/.




&)

'







&

 


!
"

'




$&

&

&

'
(




!
"

 (

'

!
"

'

!
"

 


 


&

&

 





(




'

$& #
%

!
"




 














Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.34/47

@B

U
D



 

= <
>





 @B




 @B


 2

1

 0



'

!
"








&

'

'

!
) "







!
) "

&

'

!
) "

(





&)

'

!
"
#











$&

&

&

'
(




!
"

 (

'

!
"

'




!
"

 




 


(

 


 


&

&

'

(




$&

!
"
#

(



1

 

 











Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.35/47








 (

 


 






/.

(




(

 


(

(













 





(





(

 










Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.36/47

1


2

 

 


+


 



@B
A
?


D




1 

8:



' &

1 2

1


 

5 3
7

$


 

1 

1





43

1

*


 





B H


1

GF
?



 T

( 




1 

5 3
6


1




$


 



!

1 



8: #




' &

1 2

1

 

43

!

1

"


 





B H


1

GF


;

1








1



 

I




1 

1 2

 

 I 653


G
9
8: 7

J G

G


 I I

J G



43


 

1 















Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.37/47





1


1 2

1 

 



&








 

1




U

1 2

1 

1 



1 2

1 

 






&



 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.38/47

1 


1 2




&

1 2

 

1 0

1 0






 

( 1

 



1 2





 1 

'



'






 

&


1 2

 






1




1



 


 




1 

1 


1 2


 




1 2



'



 

 




1 2

1







1




1 

&

&

 

1 

 2






1 

 2







Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.39/47

&


1 2

B

 

: 

: 

 

 




@B


2

1

&

1 2

 

(

U




&



 2

 





&

<G

 2

 





 


U




&



SG

 2

 





HR







 

1 




 

 


1




1

 

'


 





 

 


1


 



 




1


1

 

'




 

 



 


Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.40/47

1

1




 

1


1 2

1 2

K 



@B

@B

@B

@B

 @B


= <
>

!


&



!






&



'



 




Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.41/47


 




 

1

1 




 




1 0

1 0

 









 

1






Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.42/47
U

'

 

1 


 


1 2

 





'

 

 

 

 

1





1 

1


 



1





 


&

&

&



1


 



 






 

1 2


 





  

1 2


1 2






Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.43/47



1






%



 

1


1 2


 

 


( 





(



1

&




 

 

&

&

1 

1

&

&
I



 2

1 



 



1 2

1 2

1 2

;
=




1 2

1



1 




1




Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.44/47



&

 

1




1 2

 

'


1 2




 M




1

'




N






 M

'

1


1 2



1
 

1 2

1



@

IG


M

@


;



1 2









1





 


&

&

&






 

'
1

 2



 2


 

1 


1 2

1





 

1

1 2



1 2

 




 

1

1 2

1 2

 

 



FR

non-face
face

EER
FA

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.45/47






 

1 




1 2





 2






Lectures on Artificial Neural Networks Sebastien
Marcel, 2004 p.46/47



1

1





1 

'


1 

1 



 

1


 2




(

1



'

1 0

1 0













1

%

1 

1

1

N


1 

1 2

1 0

1 0

1 


N

"


1 2



1 2

(

"


 

1

1

 

1 0

1 0

 2




1

&




1

http://www.idiap.ch/marcel

http://www.torch.ch

http://www.iro.umontreal.ca/pift6266/A03

http://www.idiap.ch/bengio/lectures/index.html
http://www.cs.berkeley.edu/jordan/courses.html

Lectures on Artificial Neural Networks Sebastien

Marcel, 2004 p.47/47