Iwan Sofana (iwansofana@gmail.

com)

Kota Baru Parahyangan

Bandung West Java Indonesia
2017
 This slides adapted from many related
slides, papers, & books.

Special thanks to:

James Kennedy & Russell Eberhart
Maurice Clerc
Riccardo Poli, Tim Blackwell, Andry Pinto, Hugo Alves
Ins Domingues, Lus Rocha, Susana Cruz
Jaco F. Schutte
Matthew Settles
Satyobroto Talukder
and many more
 Optimization

Basic PSO

PSO Algorithm
Optimization
 Optimization
Optimization determines the best-suited solution to
a problem under given circumstances.

Non Linear Optimization problems are generally

very difficult to solve.

Linear/Non Linear problems are equivalent with

Linear/Non Linear function.
Function Example
 Case study
Please find the maxima/minima of the following
functions:
f(x)=x2+2
h(t)=3 + 14t 5t2
g(y)=5y3 + 2x2 3x
How do we know it is a minimum or maximum?
Clue: use derivatives & math tools.
 Case study
Quadratic

Solution
 Case study
Derivative
 Case study
Derivative rules
 Optimization
Base on problem charecteristics:

Unconstrained Optimization.

Constrained Optimization.

Dynamic Optimization.
 Unconstrained Optimization

Many optimization problems place no restrictions on

the values of that can be assigned to variables of
the problem.
Constrained Optimization
The process of optimizing an objective function with
respect to some variables in the presence of
constraints on those variables.
 Dynamic Optimization
Many optimization problems have objective
functions that change over time and such
changes in objective function cause changes in
the position of optima.
Optimization Techniques
There are two types of optimization techniques.

Global optimization technique

Local optimization technique

 Global Opt. Technique
Global optimization technique seek to find a global
minimum or lowest function value and its
corresponding global minimizer

Global minimizer (X*)

 Local Opt. Technique
Local optimization technique try to find a local
minimum and its corresponding local minimizer

Local minimizer (XL*)

Case Study

XL*

X*
 Real-Life

For a known (differentiable) function f , calculus

can fairly easily provide us with the minima and
maxima of f .

However, in real-life optimization tasks, this

objective function f is often not directly known.

The objective function is a black box.

Basic PSO
 Inspired from the nature social behavior and
dynamic movements with communications of
insects, birds and fish
 In 1986, Craig Reynolds (a biologist) studied the
flocking behavior of birds.

He described this process in 3 simple behaviors:

Separation Alignment Cohesion

avoid crowding local move towards the move toward the
flockmates average heading of average position of
(neighbors) local flockmates local flockmates
(neighbors) (neighbors)
 In 1990, Heppner and Grenander: research of bird
flocks searching for corn.

In 1995, James Kennedy (a social psychologist) &

Russell Eberhart (an electrical engineer):
influenced by Heppner and Grenanders work
developed a powerful optimization method Particle
Swarm Optimization (PSO).

I believe there are many more researcher... :-)

 PSO Origins

Kennedy and Eberhart first introduce The

Particle Swarm Optimization (PSO) algorithm for
a solution to the complex non-linear optimization
problem by imitating the behavior of bird flocks.

The Particle Swarm Optimization (PSO)

algorithm is a multi-agent parallel search
technique which maintains a swarm of particles
and each particle represents a potential solution
in the swarm.
 Basic Idea

All particles fly through a multidimensional

search space where each particle is adjusting its
position according to its own experience and that
of neighbors.

Each particle adjusts its travelling

speed dynamically corresponding to
the flying experiences of itself and its
neighbors.
PSO Algorithm
 PSO Algorithm

Basically, there are two type of PSO algorithm:

+ Local Best (lbest) PSO.
+ Global Best (gbest)

They have been developed which differ in the

size of their neighborhoods.
 Local vs Global
The local best or personal best PSO (lbest/pbest
PSO) method only allows each particle to be
influenced by the best-fit particle chosen from
its neighborhood.

The global best PSO (or gbest PSO) is a method

where the position of each particle is influenced
by the best-fit particle in the entire swarm.
 Local vs Global

The larger particle interconnectivity of the

gbest PSO, sometimes it converges faster than
the lbest PSO.

Another is due to the larger diversity of the

lbest PSO, it is less susceptible to being
trapped in local minima.
 Local vs Global
Local best or personal best PSO (lbest/pbest
PSO) use ring social topology/structure.

The global best PSO (or gbest PSO) use star

social network topology/structure.

ring social topology star social topology

local
global
 PSO Algorithm
The PSO algorithm consists of just three steps :

1. Evaluate the fitness of each particle.

2. Update individual and global best fitnesses
and positions.
3. Update velocity and position of each particle.

which are repeated until some stopping condition

is met.
 PSO Algorithm
PSO algorithm :

. Gbest PSO

. Lbest PSO
PSO Algorithm
 Uniform Distribution

A uniform distribution (a rectangular

distribution), is a distribution where the
probability of occurrence is the same for all
values. It has constant probability.

For instance, if a die is thrown, then the

probability of obtaining any one of the six
possible outcomes is 1/6.
Uniform Distribution
Geometrical Illustration
Geometrical Illustration
 Charecteristic of PSO
1. Number of particles usually between 20-60 or 10-
50.

2. C1 is the importance of personal best value.

3. C2 is the importance of neighborhood best value

Usually C1 + C2 = 4 or C1 = C2 = 2 (empirically chosen value). Wrong

initialization of C1 and C2 may result in divergent or cyclic behavior

4. If velocity is too low algorithm too slow.

5. If velocity is too high algorithm too unstable.

 Charecteristic of PSO
6. When C1,= C2 = 0 then all particles continue flying
at their current speed.

7.When C1 > 0 and C2 = 0 then all particles are

independent.

8. C2 > 0 and C1 = 0 then all particles are attracked

to single point (i.e Gbest).

9. When C1 = C2 then all particles attracked towards

the average Pbest and Gbest.
 Charecteristic of PSO
10. When C1 >> C2 each particle is more strongly
influenced by its Pbest position, resulting in excessive
wandering.

11.When C2 >> C1 and C2 = 0 then all particles are

much more influenced by the global best position,
which causes all particles to run prematurely to the
optima.
Charecteristic of PSO
Diversification Intensification

Inertia (memory) Personal influence Social influence

 Charecteristic of PSO
+ Intensification:

explores the previous solutions, finds the best

solution of a given region

+ Diversification:

searches new solutions, finds the regions with

potentially the best solutions
 Charecteristic of PSO
Advantages
Insensitive to scaling of design variables
Simple implementation
Easily parallelized for concurrent processing
Derivative free
Very few algorithm parameters
Very efficient global search algorithm

Disadvantages
Tendency to a fast and premature convergence in mid
optimum points
Slow convergence in refined search stage (weak local search
ability)
 Case study
Please find the maxima/minima of the function:
f(x)= -x2+5x +20 ; -10 <= x <= 10
Use 9 particles :

X1 = -9,6 X2 = -6

X3 = -2,6 X4 = -1,1

X5 = 0,6 X6 = 2,3

X7 = 2,8 X8 = 8,3 X9 = 10

Modified PSO
 Several approaches
2-D Otsu PSO
Active Target PSO
Adaptive PSO
Adaptive Mutation PSO
Adaptive PSO Guided by Acceleration Information
Attractive Repulsive Particle Swarm Optimization
Binary PSO
Cooperative Multiple PSO
Dynamic and Adjustable PSO
Extended Particle Swarms

Davoud Sedighizadeh and Ellips Masehian, Particle Swarm Optimization Methods, Taxonomy and
Applications. International Journal of Computer Theory and Engineering, Vol. 1, No. 5, December 2009
 Key Concepts
+ Particle Swarm Optimization (PSO) and Ant Colony
Optimization (ACO) are part of Swarm Intelligence
(SI).

+ Swarm intelligence (SI) is the collective behavior

of decentralized, self-organized systems, natural or
artificial.
 Key Concepts
1. PSO algorithm basically learned from animals
activity or behavior to solve optimization problems.

2. Each member of the population is called a particle

and the population is called a swarm.

3. It does not require any gradient information of the

function to be optimized and uses only primitive
mathematical operators.
 Key Concepts
4. PSO is well suited to solve the non-linear, non-convex,
continuous, discrete, integer variable type problems.

5. In PSO, each particle flies through the

multidimensional space and adjusts its position in every
step with its own experience and that of peers toward an
optimum solution by the entire swarm.

6. It doesnt always work well, still has room for

improvement.