Académique Documents
Professionnel Documents
Culture Documents
Performance Surfaces
1
8 Taylor Series Expansion
d ( )
F ( x ) = F ( x ) + F x ( x x )
dx x = x
2
1 d ( ) ( x x ) +
2
+ --- F x
2 d x2
x = x
n
1 d ( ) ( x x ) n +
+ ----- F x
n! d x n
x = x
2
8 Example
x
F( x) = e
x 0 0 1 0 2 1 0 3
F( x ) = e = e e ( x 0 ) + --- e ( x 0 ) --- e ( x 0 ) +
2 6
1 2 1 3
F ( x ) = 1 x + --- x --- x +
2 6
F( x ) F1( x ) = 1 x
1 2
F ( x ) F 2 ( x ) = 1 x + --- x
2
3
8 Plot of Approximations
F2 ( x )
3
2 F1 ( x )
1
F0 ( x )
-2 -1 0 1 2
4
8 Vector Case
F ( x ) = F ( x 1, x 2, , x n )
F ( x ) = F ( x ) + F(x ) ( x 1 x 1 ) + F(x) ( x 2 x 2 )
x1 x=x x 2 x=x
2
1
++ F( x ) ( x x ) + --- F ( x ) ( x x )2
xn x = x
n n 2 x2 x = x
1 1
1
2
1 ( x 1 x 1 ) ( x 2 x 2 ) +
+ --- F(x )
2 x 1 x 2 x=x
5
8 Matrix Form
F ( x ) = F ( x ) + F ( x ) ( x x )
T
x=x
1
+ --- ( x x ) 2F ( x ) ( x x ) +
T
2
x=x
Gradient Hessian
2 2 2
F(x) F( x ) F(x )
F(x) x 21 x 1 x 2 x 1 x n
x1
2 2 2
F(x) F( x) F(x ) F(x )
F ( x ) = x 2 2 F ( x ) = x 2 x 1 x 22 x 2 x n
F(x)
2
2
2
xn F( x) F( x ) F(x )
x n x 1 x n x 2 x 2n
6
8 Directional Derivatives
T
p F ( x )
First derivative (slope) of F(x) along vector p: -----------------------
p
T
Second derivative (curvature) of F(x) along vector p: p 2 F ( x ) p
------------------------------
2
p
7
8 Example
2 2
F ( x ) = x 1 + 2x 1 x 2 + 2x 2
x = 0.5 p = 1
0 1
F(x)
x1 2x 1 + 2x 2
F ( x ) = = = 1
x = x 2x 1 + 4x 2 1
F(x)
x2 x = x
x = x
1
T 1 1
p F ( x ) 1 0
----------------------- = ------------------------ = ------- = 0
p 2
1
1
8
8 Plots
Directional
Derivatives
2
20
15
1
1.4
10
1.3
5 x2 0 1.0
0 0.5
2
1 2
-1
0.0
0 1
0
-1
x2 -2 -2
-1
x1
-2
-2 -1 0 1 2
x1
9
8 Minima
Strong Minimum
The point x* is a strong minimum of F(x) if a scalar > 0 exists,
such that F(x*) < F(x* + x) for all x such that > ||x|| > 0.
Global Minimum
The point x* is a unique global minimum of F(x) if
F(x*) < F(x* + x) for all x 0.
Weak Minimum
The point x* is a weak minimum of F(x) if it is not a strong
minimum, and a scalar > 0 exists, such that F(x*) F(x* + x)
for all x such that > ||x|| > 0.
10
8 Scalar Example
4 2 1
F ( x ) = 3x 7x --- x + 6
2
8
Strong Maximum
6
2 Strong Minimum
Global Minimum
0
-2 -1 0 1 2
11
8 Vector Example
4 2 2 2
F ( x ) = ( x 2 x 1 ) + 8x 1 x 2 x 1 + x 2 + 3 F ( x ) = ( x 1 1.5x 1 x 2 + 2x 2 )x 1
2 2
1.5
1 1
0.5
0 0
-0.5
-1 -1
-1.5
-2 -2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1 0 1 2
12 8
6
8
4
2
0 0
2 2
1 2 1 2
0 1 0 1
0 0
-1 -1
-1 -1
-2 -2 -2 -2
12
8 First-Order Optimality Condition
1 T 2 ( )
F ( x ) = F ( x + x ) = F ( x ) + F ( x ) x +
T
x + --- x F x
x=x 2 x=x
x = x x
If F ( x )
T
x > 0 then F ( x x ) F ( x ) F ( x ) T x < F ( x )
x=x x = x
T
But this would imply that x* is not a minimum. Therefore F ( x ) x = 0
x = x
Therefore the Hessian matrix must be positive definite. A matrix A is positive definite if:
T
z Az > 0 for any z 0.
2x 1 + 2x 2 + 1
F ( x ) = = 0 x = 1
2x 1 + 4x 2 0.5
(Not a function of x
2F ( x ) = 2 2
2 4 in this case.)
To test the definiteness, check the eigenvalues of the Hessian. If the eigenvalues
are all greater than zero, the Hessian is positive definite.
2F ( x ) I = 2 2 2
= 6 + 4 = ( 0.76 ) ( 5.24 )
2 4
1 0 0
T 0 2 0
A' = [ B AB ] = = A = BB T
0 0 n
17
8 Second Directional Derivative
T T
p 2 F ( x ) p p Ap
------------------------------ = ---------------
2 2
p p
i c 2i
p Ap c B ( B B ) Bc c c i = 1
T T T T T
---------------
2
= -------------------------------------------
T T
- = -------------
T
- = -------------------
n
-
p c B Bc c c
c2i
i=1
T
p Ap
min --------------- max
2
p
18
8 Eigenvector (Largest Eigenvalue)
0
0
p = z max T T
c = B p = B zmax = 0
1
0
0
n
T
z max Az max
i c 2i
z1
-------------------------------
2
=1
n
- = max
- = i------------------- z2 (min)
z max
c 2i (max)
i=1
2F ( x ) = 2 0 1 = 2 z1 = 1 2 = 2 z2 = 0
02 0 1
2
0
0
2 -1
1 2
0 1
0
-1
-1
-2 -2
-2 -2 -1 0 1 2
20
8 Elliptical Hollow
1 T
F ( x ) = x 1 + x 1 x 2 + x 2 = --- x 2 1 x
2 2
2 1 2
2F ( x ) = 2 1 1 = 1 z1 = 1 2 = 3 z2 = 1
1 2 1 1
0
1
0
2 -1
1 2
0 1
0
-1
-1
-2 -2
-2 -2 -1 0 1 2
21
8 Elongated Saddle
1 2 3 1 2 1 T
F ( x ) = --- x 1 --- x 1 x 2 --- x 2 = --- x 0.5 1.5 x
4 2 4 2 1.5 0.5
2F ( x ) = 0.5 1.5 1 = 1 z1 = 1 2 = 2 z2 = 1
1.5 0.5 1 1
-4 0
-8
2
-1
1 2
0 1
0
-1
-1
-2 -2 -2
-2 -1 0 1 2
22
8 Stationary Valley
1 2 1 2 1 T
F( x ) = --- x 1 x 1 x 2 + --- x 2 = --- x 1 1 x
2 2 2 1 1
2F ( x ) = 1 1 1 = 1 z1 = 1 2 = 0 z2 = 1
1 1 1 1
0
1
0
2 -1
1 2
0 1
0
-1
-1
-2 -2
-2 -2 -1 0 1 2
23
8 Quadratic Function Summary
If the eigenvalues of the Hessian matrix are all positive, the
function will have a single strong minimum.
If the eigenvalues are all negative, the function will have a
single strong maximum.
If some eigenvalues are positive and other eigenvalues are
negative, the function will have a single saddle point.
If the eigenvalues are all nonnegative, but some
eigenvalues are zero, then the function will either have a
weak minimum or will have no stationary point.
If the eigenvalues are all nonpositive, but some
eigenvalues are zero, then the function will either have a
weak maximum or will have no stationary point.
x = A d
1
Stationary Point:
24