Benchmarking of Heuristic Optimization

Methods

Alexandr tefek
*

*
University of Defence, Faculty of Military Technology, Brno, Czech Republic, e-mail: alexandr.stefek@unob.cz


Abstract It is short history when many heuristic
optimization methods appear. As example Particle swarm
optimization method (PSO) or Repulsive particle swarm
optimization method (RPSO), Gravitational search
algorithm (GSA), Central force optimization (CFO),
Harmony search algorithm (HAS) etc. Those methods are
working differently but all of them can optimize same
problems. There is general question: Exists any standard
benchmark which can be used for individual methods
comparing? It is a bit hard to answer this question because
it is possible to find some optimization problems which are
widely used along some papers but in fact there does not
exists summary which can be uses for standard evaluation
of optimization process.
This paper will describe set of benchmarks which can be
used for evaluation of optimization process. As a part of
paper some results of optimization and utilization of
optimization methods will be included.
Keywords Heuristic, Optimization, Benchamarks.
I. INTRODUCTION
The optimization problem can be defined as:
xeR
n
,
y = (x), (1)
yeR.
Basically the problems can be divided into unimodal
and multimodal. Another aspect are constrains. Some
problems have not any constrain. The best optimization
method has to find optimal value in all cases.
Unfortunately this is, in current state of art, impossible.
Still it will be useful to be able compare different
optimization methods.

g(x) u, (2)
b(x) = u.

If the problem has defined constrain b(x) = u it can be
extended to
b(x) + z u,
z u, (S)
y = (x) + cz.
where c is coefficient which have force solver to find
optimum z = u.
There exist many algorithms for problem solving. Some
of them are special, some are more general. Many
problems cannot be solved by deterministic algorithm so
heuristic algorithm are used. In the set of heuristic
algorithm can be found PSO [1], GSA [2], CFO [3],
HAS[4]. New methods are compared to old known
solutions but there is not defined standard for solvers
(algorithms) testing.
At current state there is not possibility to reconstruct
presented solution. Very often there is not possible to use
already computed results because there does not exist
standards for measuring.
This standard measuring benchmarks can be
defined via functions. To measure different
algorithm/solvers on different problems standard problems
and interfaces to those problems must be defined.
II. UNIMODAL PROBLEMS
Unimodal functions have only one local optimum.
Those functions are relatively easy to analyze for
optimums. They are used for checking the speed of
optimization and convergence. There are used commonly
two functions. First one is Sphere and second one is Sum
and Product.
A. F1 sphere
Sphere function in fact is not sphere but the projection
to R
2
subspace will create circles around optimal solution.
This function is generalized to R
n

(x) = x

2 n
=1
. (4)
Optimum is x

= u, (x) = u
[X,Y] = meshgrid(-100:5:100,-100:5:100);
Z = X.^2 + Y.^2;
Code 1. Sphere function in 2D
Code 1 is presenting the code for use in Matlab. The R
2

version of function is show on figures below.

Fig. 1. Sphere function in 2D

8
0
0
0
8000 8000
8
0
0
0
8
0
0
0
8000
8
0
00
8
0
0
0
4
0
0
0
4000
4
0
0
0
4
0
0
0
4000
4
0
0
0
1
2
0
0
0
1
2
0
0
0
1
2
0
0
0
1
2
0
0
0
1
6
0
0
0
1
6
0
0
0
1
6
0
0
0
1
6
0
0
0
-100 -80 -60 -40 -20 0 20 40 60 80 100
-100
-80
-60
-40
-20
0
20
40
60
80
100

Fig. 2. Sphere function in 3D
B. F2 sum and product
Sum and Product function is simply to evaluate. This
function has same role as Sphere function but the different
shape.
(x) = |x

|
n
=1
+ |x

|
n
=1
. (S)
Optimum is x

= u, (x) = u
[X,Y] = meshgrid(-10:0.5:10,-10:0.5:10);
Z = abs(X).*abs(Y) + abs(X) + abs(Y);
Code 2. Sum and Product function in 2D


Fig. 3. Sum and Product function in 2D

Fig. 4. Sum and Product function in 3D

III. MULTIMODAL PROBLEMS
Multimodal functions have multiple local optimums.
Some methods will stuck in local optimum. Main goal of
those problems is to test solvers how they are able to
avoid local optimum. Some problems have no single
global optimum. Some of them have one global optimum
and many local which are very close in the term of fitness
function.

A. Schefels function
Schefels function is very often used problem for
solvers benchmarking. This problem is one of most
difficult. This problem is a bit different from others
because in some point of view it is constrained. Presented
optimum (see below) is not global optimum. This function
has no global optimum.
(x) = - x

sin|x

|
n
=1
. (6)
Commonly declared optimum is x

= 42u.96874S,
(x) = 418.9829n at interval x

e (-Suu; Suu).
[X,Y] = meshgrid(-500:10:500,-500:10:500);
Z = -X.*sin(sqrt(abs(X))) -
Y.*sin(sqrt(abs(Y)));
Code 3. Schefels function


Fig. 5. Schefels function in 2D

Fig. 6. Schefels function in 3D
4
0
4
0
4
0
40
4
0
40
4
0
4
0
8
0
8
0
8
0
8
0
-10 -8 -6 -4 -2 0 2 4 6 8 10
-10
-8
-6
-4
-2
0
2
4
6
8
10
8
0
0
80
0
8
0
0
8
0
0
8
0
0
800
8
0
0 800
8
0
0
8
0
0
8
0
0
800
800
8
0
0
800
8
0
0
8
0
0
8
0
0
8
0
0
800
800
8
0
0
800
8
0
0
80
0
8
0
0
8
0
0
800
800
8
0
0
8
0
0
1
2
00
400
1200
1
2
0
0
400
400
12
00
1200
800
8
0
0
1
2
0
0
1200
1
2
0
0
1
2
0
0
1
2
0
0
4
0
0
400
4
0
0
1200
1
2
0
0
4
0
0
4
0
0
1600
400
4
0
0
1
2
0
0
1
2
0
0
1200
40
0
400
1
2
0
0
-500 -400 -300 -200 -100 0 100 200 300 400 500
-500
-400
-300
-200
-100
0
100
200
300
400
500
Schefels function is very similar to constrained
problems. This is demonstrated on next figure where is
used only one dimension. This function has only local
optimums. Global optimum does not exist. But usually
this is not taken on mind. It is very common to test local
optimums on set (-Suu; Suu)
n


B. Rastrigins function
Rastrigins function is hills and valleys function. To
leave local optimum the mass points have to putt big
effort in point of mechanical energy.
(x) = (x

2
- 1ucos(2nx

) + 1u)
n
=1
. (7)
0ptimum is x

= u, (x) = u.
Interval x e (-S.12; S.12)
[X,Y] = meshgrid(-5.12:0.1:5.12,-5.12:0.1:5.12);
Z = 20 + X .* X + Y .* Y -10 * cos(2 * pi * X) -
10 * cos(2 * pi * Y);
Code 4. Rastrigins function

Fig. 7. Rastrigins function in 3D

Fig. 8. Rastrigins function in 3D
C. Griewanks function
Grienwanks function is function which has many
similar local optimums but there is only one global
optimum.
(x) = [
x
i
2
4000

n
=1
- cos [
x
i

n
=1
+ 1. (8)

Optimum is x

= u, (x) = u
Expected interval for solving is x

e (-6uu; 6uu)
[X,Y] = meshgrid(-600:10:600,-600:10:600);
Z = (X .* X + Y .* Y) / 4000 -cos(X) .* cos(Y /
sqrt(2)) + 1;
Code 5. Grienwanks function



Fig. 9. Grienwanks function in 3D

Fig. 10. Grienwanks function in 3D

IV. PROBLEM TRANSFORMATIONS
All but Schefels function are centered. Global
optimum is at x

= u. Heuristic optimization methods use

random initialization. Centre of particles is at x

= u
where is global optimum too. If the particles will just
move toward the centre they will reach global optimum.
This problem can be avoided by shifting.
g(x) = (x + x
s
). (9)
For common solvers (PSO, GSA, CFO) the shifting is
granting harder problem. This is basic reason why
Schefels function is too hard for some problems.
Experiments where the Schefels function was shifted so
expected optimum was at x

= u proved that problem was

more easy to solve.
4
0
4
0
4
0 4
0
4
0
4
0
4
0
4
0
4
0
4
0
4
0
4
0
40
4
0
4
0
4
0 4
0
4
0
60
60
20 2
0
2
0
20
20
2
0
2
0
2
0
2
0
20
2
0
20
20
40
40
4
0
40
4
0
40
4
0
4
0
20
20
2
0
2
0
20
2
0
2
0
20
4
0
4
0
4
0
4
0
4
0
4
0
40
4
0
6
0
6
0
6
0
60
6
0
60
60
60
4
0
4
0
40
4
0
4
0
4
0
4
0
40
40
4
0
40
4
0
2
0
2
0
2
0
2
0
2
0
2
0
2
0
2
0
20
2
0
20
2
0
20
2
0
2
0
20
40
40
40
4
0
40
4
0
4
0
4
0
40
40
4
0
4
0
20
2
0
2
0
2
0
2
0
2
0
20
20
6
0
6
0
6
0
6
0
6
0
60
60
6
0
4
0
4
0
4
0
4
0
4
0
4
0
4
0
4
0
6
0
60
6
0
60
4
0
4
0
40
40
20
2
0
2
0
2
0
20
20
2
0
20
20
2
0
2
0
2
0
6
0
60
6
0
60
6
0
6
0
6
0
6
0
4
0
4
0
4
0
4
0
4
0
4
0
4
0
4
0
2
0
2
0
2
0
2
0
-5 -4 -3 -2 -1 0 1 2 3 4 5
-5
-4
-3
-2
-1
0
1
2
3
4
5
0
.
4
0
.4
0
.
4
0
.
4
0
.
4
0
.
4
0.4
0.8
0.8 0.8
0
.
8
0
.
8
0
.
8
0
.
8
0
.8
0
.
8
0
.8
1.2
1.2
1
.2
1.2
1
.2
1
.
2
1
.2
1
.
2
1
.2
1
.
2
1
.2
1
.
2
1.2
1
.2
1.2
1
.
2
0
.8
0.8
0.8
0
.
8
0.8
0
.
8
0.8
0
.
8
0.8
0
.
8
0.8
0
.8 0.8
0
.
8
1
.
2
1
.
2
1.2
1
.2
1
.2
1
.
2
1.2
1
.
2
1.2
1
.6
1.6
1.6
1.6
1
.6
1
.6
1.6
1
.
6
1
.
6
1
.6
1
.6
0.4
0
.4
0
.4
0
.4
0
.4
0
.4
0
.4
1
.6
1
.6
1
.
6
1
.
6
1.6
1.6
2
2
2
2
2
2
2 2
2 2
2
2 2
2
2
2
-10 -8 -6 -4 -2 0 2 4 6 8 10
-10
-8
-6
-4
-2
0
2
4
6
8
10
Another possibility is usage of rotation in two
dimensions. Assume that xeR
2
then
g(x) = [j
coso sino
-sino coso
[ x. (1u)
Where o is rotation angle. The rotation removes
variables symmetry.
V. PROBLEM STANDARDIZATION
In process of optimization suite some standard
interfaces was defined. This definition extends older one
[5].
public interface IProblem
{
double Evaluate(Vector Solution);
Vector RandomSolution();
}
Code 6. Interface for problem definition
IProblem interface grants standard communication
between solver and problem.
public interface IProblemBenchmark
{
void ResetCounter();
int Counter();
float OptimalValue();
Vector OptimalSolution();
}
Code 7. Interface for problem benchmark definition
IProblemBenchmark interface allows to check the test
suite if global optimum was found (see OptimalValue and
OptimalSolution methods). Counter and ResetCounter
works with internal counter. This allows test how many
times the IProblem.Evaluate was used.
VI. SOLVER EFFORT EVALUATION
As indices which are evaluating solver effort can be
Number of calls (see IProblemBenchmark.
Counter)
Distance between optimal solution and
founded solution (IProblemBenchmark.
OptimalSolution)
Error difference between optimal value and
founded optimal value (IProblemBenchmark.
OptimalValue).
t can be
TABLE I.
DIFFERENT SOLVERS SCHEFELS FUNCTION
Solv
er
Solution Value Calls
PSO
[404,936191082104 -
348,001256167839]
-454,519415512 100 000
PSO
[-320,933366236779 -
274,460865333133]
-466,012293604 100 000
PSO
[426,686165773184
388,597262333762]
-710,190054207 100 000
PSO
[399,988156684249 -
321,41296485172]
-621,139462792 100 000
PSO
[388,522666176303
439,975273703124]
-668,679885408 100 000
GSA
[401,055625861613
432,045095352901]
-773,8611299 100 000
GSA
[-285,281251836546
433,000921879768]
-664,784988061 100 000
GSA
[-271,516959516475
208,403625506034]
-388,130741666 100 000
GSA
[-296,037442490502
209,39712069017]
-493,115686933 100 000
GSA
[-279,277321172739 -
276,313479757881]
-454,497766327 100 000
HSA
[420,973450374312
420,94526110261]
-837,965702155 100 000
HSA
[420,986577366007
420,96051663205]
-837,965725879 100 000
HSA
[420,985058378887
420,968488287632]
-837,965740961 100 000
HSA
[420,947748199547
420,967596546266]
-837,965718741 100 000
HSA
[420,993900821076
420,953676533398]
-837,965666045 100 000

VII. CONCLUSION
The problem of comparing different solvers for
optimization problems have to be solved so new methods
can be lead in already used set of solvers. The set of
problems and standard interface to interaction with those
problems will help noticeable.
This paper describes some problems which can be used
as standard benchmarks and introduced interface which
have to be implemented in testing process.

REFERENCES
[1] James Kennedy, Russell Eberhart, Particle Swarm Optimization,
Proc. IEEE Int'l. Conf. on Neural Networks, pages 19421948,
Piscataway, NJ, USA, 1995. IEEE Press.
[2] Esmat Rashedi, Hossein Nezamabadi-pour *, Saeid Saryazdi,
GSA: A Gravitational Search Algorithm, Information Sciences
179 (2009), pages 22322248, doi:10.1016/j.ins.2009.03.004
[3] R. A. Formato, Central Force Optimization: A New
Metaheuristic With Applications In Applied Electromagnetics
Progress In Electromagnetics Research, PIER 77, pages 425491,
2007
[4] Kang Seok Lee, Zong Woo Geem, A new meta-heuristic
algorithm for continuous engineering optimization: harmony
search theory and practice Comput. Methods Appl. Mech. Engrg.
194 (2005), pages 39023933, doi:10.1016/j.cma.2004.09.007
[5] Alexandr tefek, Method Cooperation in Optimization Process,
In Proc. of Mechatronika 2010, ISBN 978-80-8075-394-8
[6] Wagner , J., Maga, D., Fhrichov, R., Guba, R., Opatern., J.
Determination of step motor parameters with magnetic field
solution, Compel The International Journal for Computation
and Mathematics in Electrical and Electronic Engineering. 1994,
vol. 13, Nr. 1, pp. 137-139. ISSN 0332-1649.