I.

Introduction to Artificial neural networks

1.1. Analogy to the Brain

Artificial intelligence ( AI )
artificial neural network ( ANN )

ANN

၎
Human
brain element neuron cell

neuron
Human brain neuron 100
neuron
electro-chemical signal

(၁-၁) ၎

1 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Human brain
dendrites axons
cell body
myelinated fibres

frontal parietal
temporal occipital Frontal

frontal
frontal
Parietal

temporal
Occipital

cortex hippocampus thalamus cerebellum

amygdala striate body

cortex
information
cortex Human brain
၂

element neuron
biological neural network
biological neural network

2 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 artificial neural
network

1.2. Biological neuron and artificial neuron

1.2.1. Biological neuron

Biological neural network biological neuron

human brain element biological
neuron biological neuron
၎
dendrites cell body axon
Dendrites
neuron signal neuron
synapses dendrites
neuron data input
input signal cell body cell body
input signal ၎ cell body
neuron axon
axon neuron output signal
neuron synapses
dendrites cell body signal input
၎ cell-body element ၎
axon cell-body signal ၎
Neuron element synapses
neuron element
neuron body, dendrite, axon synapse

3 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၂) Biological neuron

1.2.2. Artificial neuron

Artificial neuron biological neuron computing

neuron
artificial neuron biological neuron
input processing element (node) output
Biological neuron
artificial neuron

(၁-၃) Artificial neuron

4 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Artificial neuron biological neuron
၎

Biological neuron Artificial neuron

Cell body Node
Dendrites Input
Axon Output
Synapse Weight
(၁-၁) Biological neuron artificial neuron

(၁-၁) artificial neuron biological neuron

element biological neuron
dendrite input ၎ axon output ၎ cell body node
synapse weight artificial neuron

1.2.3. The working principle of an artificial neuron

Artificial neuron (၁-၃)

၎ sum activation function
artificial neuron
input data weight data
neuron activation function ( activation function
(၁-၃) )
activation function input
၎ function
Artificial neuron
(၁-၃) neuron input
Artificial neuron
neuron activation function net_input input data

5 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 weight data
၎

equation vector input data

weight data vector

net_input matrix multiplication

weighted data (
) neuron activation
function input Neuron activation function
၎ input activation function
neuron activation function threshold function
Threshold function

၎ function net_input neuron

( ) neuron
1 artificial neuron processing
mathematical model

6 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 artificial neuron input data neuron
weight data neural network bias
data bias data artificial neuron (၁-၄)

(၁-၄) bias artificial neuron

Artificial neuron bias classifier

decision boundary ( ) neuron
Bias
artificial neuron mathematical model
bias ၎ mathematical model

bias neuron

7 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.2.4. The working example of an artificial neuron

Artificial neural network

artificial neuron
artificial neuron
(၁-၅) ၎ processing

(၁-၅) Artificial neuron

(၁-၅) artificial neuron input

data neuron weight
data activation function
threshold function

Neuron output input data weight data

neuron activation function

8 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
activation function net_input input data weight data

Net_input neuron activation function net_input

network output

vector

input data weight data

vector

1.3. Activation function

Neural network neuron activation function

၎ neuron activation
function activation function
Neuron input
weight activation function
input neuron activation function
(၁-၂-၃)
neuron input signal output
neuron
Neuron activation function

9 |ZawMinKhaing@MIET51 @ F B / Z a w M i n K h a i n g 1 9 8 9
 ၎ function activation function
(၁-၂

Name Formula Range

Linear
Semilinear

Logistic (sigmoidal)

Hyperbolic tangent (sigmoidal)

Exponential
Sinusoidal (sine)
Rational (sigmoidal)

Step

Hard limit (threshold)

Modular
Signed (signature)

Quadratic
Relu
(၁-၂) ANN

activation function function

hard-limit linear sigmoid function Hard-limit function
function input neuron output
input 0 ( ) 0 neuron output
neuron output 0 1
Hard-limit function graph

10 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၆) Hard-limit

grap input hard-limit

function input
input
Linear function neuron input

၎ function graph

(၁-၇) Linear

Hard-limit linear function neuron function

sigmoid non-linear function (logistic function
) function
၎ function neuron output

11 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 ၎ function graph

(၁-၈) Log-Sigmoid

၎ sigmoid function
input

1.4. Artificial neural network

Artificial neural network (ANN) human brain

biological neuron biological neural network
network
model artificial
neuron network
Artificial neuron network ၎ neuron
(layers) neuron
network model Artificial neuron
artificial neural network
neural network architecture

12 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.5. Classification of artificial neural network

Neural network (feed-

forward) network
(feedback)
artificl neural network classification ၎
network feedback /
feed-forward feedback class Feedback
neural network statics networks
feedback network recurrent networks ( dynamic networks )
(၁-၉) artificial neural network
class

(၁-၉) Artificial neural network

13 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Feedback / class ၎ topology
neural network Topology fully
connected neural network, unfully connected neural network multilayer neural
networks
Fully connected network neuron ၎ output signal
neuron Network
neuron neuron
network Hopfield network
Unfully connected neural network neuron
neurons neurons
Multi-layer network neuron
layers network
neural network ၎
neuron
network layer
Neural network neuron homogeneous
heterogeneous class Network neuron
activation function ၎ network
homogeneous network neuron activation function
၎ network heterogeneous network

network ( )
network network
network hybrid neural network network
multi-layer network
topology ၎ network
learning algorithm

14 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Neural network
problem network
Problem network topology
developer problem
universal neural network
( neural network

network network
)

1.6. Single layer artificial neural network (Perceptron)

artificial neural network

၎ single layer neural network
neural network model ၎ network
perceptron Single layer neural network
neuron

(၁-၁ ) Perceptron neural network

15 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၁ ) single layer neural network
၎ input signal
၎ neuron Perceptron network
neuron (၁-၁ )
neural network input neuron
၎ neuron
Network neuron
network neuron output

Single layer neuron input signal weight

neuron
input single weight neuron_1 input
signal weight data
neuron_2 input signal weight data
Neuron
input signal neuron weight data

Network input vector

weight data vector

matrix neuron
input
network neuron input connection

input neuron
neuron
output
neuron output

16 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Network output input vector
activation activation
( )

1.6.1. The working example of perceptron

Perceptron network
(၁-၁၁) neuron
input single layer neural network output

(၁-၁၁) Perceptron neural network

17 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Network input data weight data function
network
output

Network ၎ output

network output matrix

neuron output
neuron output

neuron output
neuron output
single layer neural network linear separable
problem

18 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.7. Multi-layer neural network

Neuron neural network

Multi-layer network
network single layer network
network multi-layer
neural network perceptron network
၎
၎

(၁-၁၂) multi-layer neural network

(၁-၁၂) multi-layer network

၎
input Multi-layer network
input hidden output
Input
input data weight data hidden
neurons input data Hidden input
input data weight data

19 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
Multi-layer hidden ၎ hidden
network output
hidden

1.7.1. The working principle of multi-layer neural network

Multi-layer neural network weight data

weight data
multi-layer neural network
parameters

(၁-၁၃) Multi-layer neural network weight matrix

multi-layer network hidden

neuron output neuron

neuron activation function Input

hidden matrix
20 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 ၎ weight matrix single layer
network input data hidden neuron
hidden
output data hidden
hidden hidden
၎ weight matrix ၎
matrix Network output neuron
hidden output
matrix network
network
output network output

weight data matrix

output
network ၎
output Hidden
output ၎ output

Hidden output network output

network
output equation

output of the neural network

output of the hidden layer_1 and hidden layer_2
weight matrices of hidden layer_1 hidden layer_2 and output layer
activation functions of hidden layer_1 hidden layer_2 and output layer

21 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.7.2. The working example of multi-layer neural network

multi-layer neural network

universal approximator
multi-layer neural network

(၁-၁၄) Function approximator neural network

Function approximator multi-layer neural network

hidden tansig activation function output
linear activation function (၁-၁၄)

input vector hidden layer weight matrix

output layer weight matrix

(weight matrix row column

)
Hidden layer weight input data hidden layer

22 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 output layer hidden layer
output data output layer
Network output hidden layer
output ၎ activation
input data hidden layer

Hiddel layer neuron ၎

Hidden layer output
network

multi-
layer neural network output

input data set

hidden layer output neural network output
hidden layer output layer activation functions
hidden layer output layer weight matrix

23 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.8. Radial Basis Function Network (RBF)

RBF neural network feed-forward multi-layer neural

network ၎ neural network multi-
layer neural network
neural network hidden
neuron radial basis function (RBF)
၎ neural network RBF neural network
၎ network hidden neuron radial basis function
(RBF) ၎ network
multi-layer neural network input
hidden output
၎ RBF neural network

(၁-၁၅) RBF neural network

(၁-၁၅) RBF neural network input hidden

output hidden
neuron radial basis function output

24 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
neuron linear function RBF
network feature RBF function hidden layer

1.8.1. Radial basis function

RBF neural network function approximation

multi-layer neural network
၎ network classification
RBF neural network hidden
neuron radial basis function
RBF neural network radial function
(၁-၃

Function name Formula

Gaussian

Multi-quadric
Generalized Multi-Quadric
Inverse Multi-Quadric
Generalized Inverse Multi-Quadric
Thin Plate Spline
Cubic
(၁-၃) Radial basis function

Radial function Gaussian function RBF network

scalar input ၎ formula

25 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 Gaussian function formula parameter ၎ central point
radius (spread ) radial basis function
graph

(၁-၁၆) Radial basis function

Radial basis function central point spread

parameter input central point
output 0 1
(Central point output )

1.8.1. The working principle of radial basis neural network

Multi-layer neural network hidden neuron

input data weight data weighted data
input RBF neural network
RBF neural network weighted data
neuron input input data
weight data Euclidean distance neuron input
RBF neural network
network parameter

26 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၁၇) RBF network parameter

(၁-၁၇) Radial basis network hidden

neuron radial basis function central point
spread parameter Neuron input
data central point parameter input data
neuron central point parameter
hidden neuron central point
parameter matrix ၎ hidden
neuron input Hidden
neuron central point parameter spread parameter
Neuron spread
hidden layer neuron
spread parameter
၎ vector

Hidden layer operation

RBF neural network hidden neuron multi-layer
network neuron input data weight data ( central point
parameters ) network weight

27 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
data input data ၎ Euclidean distance
Euclidean distance formula

formula hidden neuron Euclidean distance

hidden Neuron Euclidean distance

neuron input weight data Euclidean distance

hidden layer neuron output
Hidden neuron activation function
Gaussian function
Euclidean distance ၎ function

hidden neuron output

hidden_layer neuron output Euclidean distance

spread parameter

Output layer operation

Network output hidden output
output neuron weight
perceptron multi-layer
neural network weighted data neuron input
hidden output vector
output output multi-layer network

28 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
RBF neural network output neuron linear function
output output neuron
RBF network

input data set

hidden layer output neural network output
output layer activation function
hidden layer output layer weight matrix
spread parameters vector

1.8.2. The working example of RBF neural network

RBF neural network

(၁-၁၈) input (၃) hidden neuron (၄) output
neuron (၂) RBF network

(၁-၁၈) RBF neural network

29 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 network input data

၎hidden layer

parameter Hidden neuron

spread parameters vector

Hidden output matrix

Hidden

input matrix Euclidean distance

Hidden neuron Euclidean distance

၎ neuron radial basis function Euclidean distance
spread parameters hidden output

30 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 hidden output ၎ weight
output activation function
output activation function feed-forward neural
network hidden output weighted data

neuron activation function

input output output
Output activation function linear function
function input data network output
RBF network output layer
linear activation function classification
hard-lim ( ) sigmoid function

31 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.9. Convolutional neural networks (CNN)

Convolutional neural network (CNN) image, speech object

recognition neural network
၎ CNN neural network
convolutional layer (C layer) subsampling layer (S layer)
output fully connected layer (F layer) network
၎ architecture

(၁-၁၉) Convolutional neural network

CNN C layer S layer F layer

local perception, share weights subsampling (pooling)
paradigm Local perception image
input data layer neuron
image Sharing
weight connection weight
neural network
weight data image [32*32*3] neuron 10

32 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 layer weight matrix [3072*10] = 30720
input image [32*32*3] [5*5*3]
weight matrix
weight sh
kernel (filter) Subsampling image
spatial dimension Input image
invariance

1.9.1. The working principle of CNN

(၁-၁၈) convolutional neural network convolutional

layer (C layer) subsampling layer (S layer) fully connected layer (F layer)
CNN
၎ layer

Convolution layer
Convolution layer CNN ၎
image
input image filter
Convolution (filter) two dimensional matrix
coefficient filter input dimensional
image output image
Element-wise multiplication
convolution process (၁-၂ ) ၎
(၁-၂၁)

33 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၂

(၁-၂၁) Image c

convolution process filter image

pixel
pixel

stride parmeter
(၁-၂၁) stride (၁ stride (၂
(၁-၂၂)

34 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၂၂) Stride (၂

(၁-၂၂) stride (၂
feature map
stride feature map
Padding image
( zero
) image pad

image padding size (၁) padding size (၂

(၁-၂၃)

(၁-၂၃) Image
35 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 filter (F) stride (S) zero
padding (P) hyperparameters image
convolution layer filter output image (feature map)

output image volume size ( feature map )

input image volume size
size of the filter
the number of zero padding
the number of stride
hyperparmeters standard

Subsamplig layer (pooling layer)

Convolution layer feature map subsampling
layer Subsampling layer
image invariance
CNN image cooridinate
၎ image feature
image feature
၎ layer neighbor maximum
( ) average maximum ( ) average
neuron image subsampling
layer feature map subsampling layer pooling
image
(၁-၂၄)

subsampling layer maximum

36 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 (၁-၂၄) Maximun pooling pr

Fully connected layer

convolution layer subsampling layer
CNN image feature
feature map
feature map vector (
CNN fully connect layer output layer

Fully connect layer CNN layer neural

network layer type layer neuron
non-linear activation function logistic function
hyperbolic tangent function
activation function Fully connect neural network multi-layer neural
network ၎
(၁.၇.၁)

37 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 1.9.2. Calculation the parameters of CNN

CNN deep neural network ၎ network layer

parameter neural network
CNN layer
filter feature map parameter
CNN parameters
LeNet-5 neural network parameter
LeNet-5 neural network
input -> conv1 -> pool1 -> conv2 -> pool2 -> conv3 ->
fully_connected -> output ၎ network C layer
S layer feature map conv1(6@28*28) pool1(6@14*14)
conv2(16@10*10) pool2(16@5*5) conv3(120@1*1) ၎ network

(၁-၂၅) LeNet neural network

38 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 LeNet ၎ network hand digit recognition
application LeNet input image 32*32
grayscale image input image
filter (conv1) layer
feature map
conv1 layer feature map
feature map
feature map ( (feature map
filter )
filter parameters
input image feature
maps parameter

Conv1 layer feature map subsampling

layer (pool1) subsampling layer feature
maps filter
parameter conv1 layer
feature map subsampling layer (pool1) feature map
parameter

Subsampling layer (pool1) feature map convolution layer

(conv2) conv2 filter conv1
၎ filter conv1 filter
C method conv2 layer
conv1 feature map filter ၎
(၁-၄)

39 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0 * * * * * * * * * *
1 * * * * * * * * * *
2 * * * * * * * * * *
3 * * * * * * * * * *
4 * * * * * * * * * *
5 * * * * * * * * * *
(၁-၄) Subsampling layer(pool1) convolution layer (conv2) feature map

column (conv2) feature map

(pool1) feature map
column (conv2) feature
map (pool1) feature map (၃)
ၸ ၎

(၁-၂၆) Subsampling layer(pool1) convolution layer (conv2) feature map

feature
connection filter (conv2) parameter

convolutional
layer (conv2) feature map subsampling layer (pool1) feature map
40 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 connection

conv2 layer feature map subsampling layer (pool2)

filter p feature maps
subsampling layer (pool2) feature map conv2
feature map parameter
1 16
Convolution layer (conv3) fearutr map
Subsampling layer (pool2) feature maps
parameter
Convolution layer (conv3) layer
neuron (conv3) fearutr map
parameter
LeNet-5 neural network parameters
(၁-၅
Input Filter Output Filter parameter Connection parameter
size size size
Conv1 32*32 6@5*5 6@28*28 (5*5+1)*6 = 156 156*28*28 = 122304

Pool1 6@28*28 1@2*2 6@14*14 (1+1)*6 = 12 (2*2+1)*6*14*14=

5880
Conv2 6@14*14 16@5*5 16@10*10 (5*5*4+1)*6+(5* 1516*10*10 = 151600
5*4+1)*3+(5*5*6
+1) = 1516
Pool2 16@10*10 1@2*2 16@5*5 (1+1)*16 = 32 (2*2+1)*16*5*5 =
2000
Conv3 16@5*5 120@5*5 120@1*1 (5*5*16+1)*120 48120*1*1 = 48120
= 48120
Fully 120 84 (120+1)*84 = 10164
connected
Classification 84 10 84*10 = 840

(၁-၅) LeNet neural network

41 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9
 LeNet-5 neural network
CNN ၎ network computer scientist
Yann LeCun (http://yann.lecun.com/) 1998
IEEE (http://yann.lecun.com/exdb/publis/pdf/lecun-
98.pdf)
LeNet-5 CNN handwritten characters variability
pattern CNN
develop ImageNet AlexNet(2012)
Clarifi(2013) VGGNet(2014) GoogLeNet(2014) Inception v2, v3 ResNet(2015)
Inception v4 v3(2016) CNN
computer science
Modern CNN classicial CNN mathematical
operation ၎ network
convolutional layer (C layer) subsampling layer (S
layer) fully connected layer (F layer)

42 | Z a w M i n K h a i n g @ M I E T 5 1 @ F B / Z a w M i n K h a i n g 1 9 8 9