Chapter 3

Mathematical concepts

Continuity of function
A function of a single variable is continuous at a point if
lim and are exist

lim =

Contour line of function

A contour of a function is a curve along which the
function has a constant value

Optimization problem statement

A general form for a problem

Minimize:
Subject to:
and:

= 1, ,
= 1, ,

In which is a vector of variables = 1 , 2 , ,

is the objective function
are constraint functions
, are lower and upper bounds on the constraints
, are lower and upper bounds on the variables

Optimization problem statement

One or more of , 1 , 2 , , is nonlinear: NLP prob.

There are no constraint function and no bound on

variables : unconstrained prob.
Only the are bounded: boundconstrained prob.
All constraint functions are linear and the objective is
nonlinear: linear constrained prob.

Linear constrained problem with a quadratic objective:

quadratic program (QP)

Optimization problem statement

A vector that satisfies all constraints is feasible vector
The set of all feasible points is the feasible region
A typical feasible region for a problem
with two variables and constraints
0
0
= 1,2
= 1,2

2 = 0

Optimization problem statement

A point (vector) is a local extremum (minimum) if

(in neighborhood)
For global extremum:

(in whole feasible region )

Almost of algorithm for solving optimization problems

locate a local extremum from a given starting point

Convexity of function
1

2
1 + 1 2

1 + 1 2
1

1 + 1 2 < 1 + 1 2

01

strictly convex

1 + 1 2 1 + 1 2
01

convex

Convexity of function
2

1
1 + 1 2

1 + 1 2
1

1 + 1 2 > 1 + 1 2

01

strictly concave

1 + 1 2 1 + 1 2
01

concave

Convexity of function
property
strictly convex
convex
strictly concave
concave

property
positivedefinite
positivesemidefinite
negativedefinite
negativesemidefinite

undefined
indefinite
convex & concave

All eigenvalues of
> 0
0
< 0

0
> 0 or < 0
= 0

Linear functions are both convex and concave,

but not strictly convex or concave

Convexity of function
Equivalent condition for Hessian matrix property
property

strictly convex
convex
strictly concave
concave
undefined
convex & concave

property

=
> 0 for all 0

positivedefinite
positivesemidefinite 0 for all 0
negativedefinite
< 0 for all 0
negativesemidefinite 0 for all 0
indefinite
> 0 or < 0
= 0 for all 0

Convexity of function
Significance of convex function
Case property
Number of extremum
1 unimodal
one minimum
2
3
4

multimodal
unimodal
multimodal

multimodal

more than one minimum

one maximum
more than one maximum
saddle point or
more than one extremum
none (no extremum)

property
strictly convex
convex
strictly concave
concave
undefined
convex & concave

Example 1
Determine the convexity of these functions in range ;

1) = 3 2
2) = 2

3) = 5 2
4) = 2 2 3

Convexity of region
A convex region in which, for all pairs of points 1 and 2
in the region, the straightline joining them is also
entirely in the region
= 1 + 1 2
01
All of , 1 and 2 in the region

Convex region may be closed or open

Convexity of region

Linear functions are both convex and concave. Therefore if

a region is bounded by 0 = 1, , in which
each constraint is linear, then the region is convex

Example 2
Determine the convexity of this region
12 + 2 1

1 2 2

1 = 12 2 + 1 0

2 = 1 + 2 2 0

Example 3
Determine the convexity of this region
1 + 2 6
0 1 6
0 2 6
1 = 1 + 2 6
2 = 1 6
3 = 2 6
4 = 1 0
5 = 2 0

Convex programming problem

For nonlinear optimization problem

Minimize:
Subject to: 0

= 1, ,

If is a convex function, and each constraint

is a convex function (so that the constraints form a
convex region), then
The local minimum of is also the global minimum

Convex programming problem

For nonconvex problem, almost of algorithm find at least

a local minimum, and it is usually impossible to prove
that a given algorithm will find the global minimum

necessary and sufficient condition

necessary condition for a minimum or maximum of
is that the gradient of vanishes at
= 0
that is, is a stationary point

necessary and sufficient condition

sufficient condition for a minimum or maximum of is
that the Hessian of is positivedefinite or negative
definite, respectively
property
positivedefinite
negativedefinite
indefinite

Minimum point
Maximum point
Saddle point

A minimum or maximum may exist at even

though these conditions are not satisfied

necessary and sufficient condition

For function of a single variable, the higher derivatives at
the stationary point, , is required to evaluate the
extremum if the sufficient condition is not satisfied
If is even:
> 0, is minimum point
< 0, is maximum point
If is odd, is saddle point

Example 4
Determine the extrema of the function
= 4
= 4 3

= 12 2

necessary condition

= 0

sufficient condition

= 0 saddle point?

= 24 > 0

or

minimum point

= 0