Optimization theory and advanced computing

methods

Proper local minimum Local minimum and global minimum of a function f(x)

∧
∧ The point x determines a local minimum of a function f(x) in Rn space,
f(x) has a proper local minimum in point x , if there exists
Function f(x) δ >0 When: there exists E ⊂ R
n
∧
for point x , that
such,
such, that for ∀x ∈ E )
f (x ) < f (x )
∧
∀x ∈ E f ( x) ≤ f ( x)

where:
where: E = X ∩∆ ∧
∧
When f ( x) < f ( x) for x ≠ x then the proper local minimum exists.
∆ = {x : 0 < x − x < δ }
)
∧
∧
Week localx minimum The point x determines a global minimum of a function f(x) in Rn space,
∧
Function f(x)
f(x) has a week local minimum in point x , if there exists δl > 0 ∧
when ∀x ∈ R n f ( x) ≤ f ( x)
such that,
that, for ∀x ∈ El ) )
f ( x )≤ f (xl )
where: ∧ ∧
When f ( x) < f ( x) for x≠x then the proper global minimum exists.
El = X ∩ ∆ l

∆ l = {x : 0 < x − x < δ l }
)

Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic

The Weierstrass extreme value theorem

Definition of a convex set

In Euclidean space the set X X ⊂ R n is convex,

convex, if for every pair of points x1 , x 2 ∈ X
A continuous function f(x) defined on a compact set of admissible solutions (on within the set, every point on the straight line segment that joins the pairs of
a closed and bounded set) points is also within this set X :

is bounded on this set and obtains its bounds.: two points exist, x1 , x2 ∈ X
{
X = x : x = λx1 + (1 − λ )x 2 , 0 ≤ λ ≤ 1 }
such that for each x∈ X

the following relation is valued: f ( x1 ) ≤ f ( x ) ≤ f ( x 2 )

Definition of a convex function

In Euclidean space the set X ( X ⊂ R n ) is convex. f(x) f : X → R1 will

convex. Function f(x)
function, if for every pair of points x1 , x 2 ∈ X
be a convex function,
and each λ ∈ [0,1] the following inequality is fulfilled:
fulfilled:

( ) ( )
f λx1 + (1 − λ )x 2 ≤ λ f x1 + (1 − λ ) f x 2 ( )
Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic

Is a function f(x) – a convex function

Definition of a strictly convex function
A matrix A (dim nxn) is called hessian, when its elements are partial
In Euclidean space the set X X ⊂ R n is convex. derivative of second order of a function f(x):
convex. Function f(x):
f(x): f : X → R1
Will be strictly convex,
convex, if for every pair of points x1 , x 2 ∈ X  ∂ 2 f ( x) 
A( x) =  
And each λ ∈ [0,1] the following inequality is fulfilled:
fulfilled:  ∂xi ∂x j 
f (λx1 + (1 − λ )x 2 ) < λ f (x1 ) + (1 − λ ) f (x 2 )
Lemma 1
Definition of a concave function
In Euclidean space the set X ( X ⊂ R ) is convex. f(x) f : X → R1 will
n
convex. Function f(x) Let f : X → R1 has continuous partial derivative with second order and let
function, if for every pair of points x1 , x 2 ∈ X
be a concave function,
X ⊂ Rn
and each λ ∈ [0,1] the following inequality is fulfilled:
fulfilled: is a convex set. f(x) is a convex function if and only if

( ) ( )
f λx1 + (1 − λ )x 2 ≥ λ f x1 + (1 − λ ) f x 2 ( ) her hessian A(x) is positive half-definite for each x∈ X .

Collorary: Function f(x) is concave if and only if function -f(x) is convex.

Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic
Optimization theory and advanced computing
methods

Definition of a positive half-definite matrix A

Sylwester Criterion –
Practical testing of a function convexity
A matrix A is a positive half-
half-definite matrix,
matrix, when for each x∈R n

x, Ax ≥ 0 A square symmetric matrix A is positive half-

half-definite matrix,
matrix, if and only if:
if:

a11 a12 ,..., a1n

a11 a12 a 21 a 22 ,..., a 2 n
a11 ≥ 0, ≥ 0 ,..., ≥0
Definition of a positive definite matrix A a 21 a22 . ... .
a n1 ... ,ann

A square symmetric matrix A is a positive definite matrix,

matrix, if and only if:
if:
A matrix A is a positive definite matrix,
matrix, when for each x ∈ Rn

a11 a12 ,..., a1n

a11 a12 a 21 a 22 ,..., a 2n
x, Ax > 0 a11 > 0, > 0 ,..., >0
a 21 a 22 . ... .
a n1 ... , ann

Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic

Fig.1 The convex set X Fig.2 The non-convex set of X

Function f(x) – a convex function.

Convex function f(x)
f(x)
Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic

Fig.3 The convex set of X Fig.4 The non-convex set of X

Function f(x) – non-convex function Function f(x) – non-convex function.

Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic
Optimization theory and advanced computing
methods

Convex function

Theorem 2 Theorem 4
Let our set X X ⊂ R n will be convex set and let function f(x) wil be differentiable.

Then function f(x) will be convex ∀x, x 0 ∈ X , if and only if Let X ⊂ R n is a convex set . When f i : X → R1 for i=1,...,k are convex
functions and if scalar value
αi ≥ 0
f ( x) ≥ f ( x 0 ) + < ∇f ( x 0 ), x − x 0 >
for i=1,...,k then function f(x)
f(x)
Function f(x) is a convex function on a range (a,b) if and only if the function diagram lays above the k
diagram of a tangent to a curve for each point x0 from a range (a,b). f ( x) = ∑α i f i (x )
i =1

Is a convex function.
Theorem 3
Let a function f : X → R1, for X ⊂ R n will be convex function, than for each real value α
the set
X α = {x : f ( x) ≤ α }
is convex.

Optimization theory and advanced Faculty of Electronics

computing methods Ph.D. Ewa Szlachcic

Theorem 5
The local minimum of a convex function on a convex set X ⊂ R n of admissible
solutions, is global minimum of that function on the set X. ( )
f x2 ≥
( )
f λx1 + (1 − λ )x 2 − λf x1
≥
( )
f x 1 − λ f x1 ( )
= f x1
( ) ( )
1− λ 1− λ
Proof:
Let the function f(x) has his local minimum in point x1 ∈ X .
which completes the proof.
It means that there exists such ε >0 , that:

( )
f x = min f ( x )
1
x∈V

where: {
V = x ∈ X , x − x1 ≤ ε } Theorem 6

Assume, that x2 ∈ X , x1 ≠ x 2 . Let us consider 0 < λ <1 and Strictly convex function f(x) defined on a convex set of admissible solutions

λx1 + (1 − λ )x 2 ∈ V has at most one minimum on this set.

In view of convexity of a function f(x) :

λ f (x1 ) + (1 − λ ) f (x 2 ) ≥ f (λx1 + (1 − λ )x 2 )

Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic

Theorem 7
Example When
Let c ∈ Rn , d ∈ R1 . Try to show, that function f(x) f ( x ) : R n → R1 f (x ) : R n → R
defined as follows: Is concave function and X is a bounded, closed and convex set,

f ( x ) = cT x + d there exists extreme point of a set X, where the function f(x) achieves the minimum
value on this set.
Is simultaneously convex and concave function on a set of Rn.

Colorary

The linear function f(x) defined on a polyhedron (the convex set of admissible
solutions) achieves it’s bounds on extreme points of a polyhedron.

Optimization theory and advanced Faculty of Electronics Optimization theory and advanced Faculty of Electronics
computing methods Ph.D. Ewa Szlachcic computing methods Ph.D. Ewa Szlachcic