" Opiimization Unconstrained

ruecessarvanO Sutl The necessary conditions for an unconstrained optimum at x* are, Vf(x.) = g at x* and f(x) be differentiable

Sutficient Conditions for a Minimum definite The aboveplusV2f(x*)is positive for a Maximum Sutficient Conditions definite The aboveplusV2f(x.)is negative Steepest Oescent a is steepest methods descent.The steepest descent One of the simplest optimization point,the algorithm moves starting afgorithm is a searching algorithm.Givenan initial downhill untilit can go no fufther. we must The searchcan be brokendown into stages. At each stage (or iteration) determine two things: 1. 2. be? Whatshouldthe searchdirection Howfar shouldwe go in that direction?

is: the searchdirection For methodof steepest descent, This answersquestion1 uLine" a "linesearch." in this case meanswe 2 is answered by conducting Question search along a direction. The line search strategypresentedhere, braketingthe and is one of the fit, is one of manythat have been proposed, functionwith quadratic by the method example most common. We will illustrate point,xk, we wishto determine Givensomestarting
a k + 1= X k + C r S

vector wheres = searchdirection cr= step length.(scalar) by its magnitude. s is normalized by dividing Usually

Unconstrained Optimization

Example

Min f: *l - z4*, * 44 startins rt = [;3] ",

f(x)= 19 -Vf(xo1 = [-?r]

normarized s1 [_BJ ,.,=[ltl + "[-.f.]

We willfind cr*, the optimal steplength,by trialand error. Guessa,*= .4 for stepno. 1: Line SearchStep
g X2=X1+ CrSl

"' =[-?r]

f(x)

1 2 s 4

. -[-3J=[-?3] 13.3 4 o=[-?] + '[-3J=[-??] 8.75 I *=[-?] =[:i3] 3.74 16 o=f?]+16t-SJ =[-;l;i] 9 . 3 1 s2 *=f?] + 32t-3J

we havegonetoo far, The objective functionhas startedto increase: We will cut the changein the laststepby half: 5 Plotting the linesearch: Line SearchExample
14 12 o 1 o 1 0 - 16.1386x + 4.0897x^2 Y = 19.0794

=[-;lo3] 24 o=f?] +24[-a3]

3.91

.
b

c) o
l

9
o

8
4 2

10-

a H ><
I

><
I
a

ftI I

lf

co
I

x
x
lL

cl

v
I

n
I

(o
N
6

tS av. \E-'/ i

, :

_ZX STXV

i i

Unconstrained Optimization

We wouldliketo estimate the minimum We see thatthe dataplotup to be a paraboia. points points 2, 5, 3. These are equallyspacedand of this curve. We will curuefit bracket the minimum. ll2 ll3 ll5 of the parabola is givenby Renumbering thesepointsas cr1 , a2, a3 the minimum - f(crs)] acr[f(cr1) a,* = a,2* z[t1a1)-2t(u2) + f(a3)] ( . 8 ) [ 8 . 75 3.e1]

cr*= 1.6 +

- 2(s.24) + o.e1] zla.zs

= 1.97 wheref(x) = 3.2 untilthe of stepping alongthe searchdirection In summary, the line searchconsists points fitting which bracket is bracketed, three this direction the function in minimum of parabola. lf the minimumof the and calculating the minimumwith a parabola, fit can be carriedout severaltimes until the changein the necessary the parabolic spaced,and the the s are then no longerequally is very small(although minimum formula following can be used):
cf,* =

f(arXai

Zft@fl(crz cs) + f(cr2)(cr3 or) + f(o3)(crr oz)]

"trl -

+ f(a2)(a?

- "1)

+ f(og)( "? - "71

Derivative The Directional e usefulto be calculat It is sometimes chainrulefor differentiation,

-df =

df s. From the alongsomesearchdirection fr

dcr vectorx, df da

1k+1 tffi,[*.J Notinsthat

xk + crs, or, for a single element of

=vf(x) '[+F:)='[*,]
Tg

= si,so *lt*t=\k* *i, wenave lf,

Unconstrained Optimization Descent Weakness of Method of Steepest is eccentric, convergence of steepestdescentcan be lf the functionto be minimized quiteslow: Prog.), Theorem (fromLuenberger, lntro.to Linearand Nonlinear descent Descent-Quadratic Case: For any xo e En the methodof steepest Steepest pointx* of f. Futhermore, with E(x)=tr* -x.)TH(x to the uniqueminimum converges x*), thereholdsat everystepk, E(xk+t,=(^*) where E(xx;

of H eigenvalue A= largest = of H l smdllest eigenvalue

rate of steepest the abovetheoremsays that the convergence "Roughly speaking, more eccentric. lf of f become a=A, corresponding descentis slowedas the contours that even if occursin a singlestep. Note,however, to circularcontours, convergence from one is a greatdistance n-1 of the n eigenvalues are equaland the remaining can these, convergence will be slow, and hence an single abnormaleigenvalue descent." destroythe effectiveness of steepest
Proiection of 9/ (r) on surfoceol /(r)

Fig. 3.1-3 Characleristic oscillarion ol rhe method of slaepest descent in a narrow valley.

Unconstrained Optimization 5

Newton's Method Anotherclassical methodwe will brieflystudy is calledNewton's method. lt is to a functionat the currentpoint and approximation simplyto make a quadratic has an optimum.Starting with a Taylor's solvefor wherethis approximation series: * ]{xx*r-xk)rH(xk)(xk+1-x$ f(xk+t)= f(xk)+ vf(xr<1r11k+1-ak; (1)

at k, they are just a vectorand matrixof are evaluated and hessian Sincethe gradient (xk+1-v!, Ax for constants.Substituting f(xk+r;= f(xk)+ vf(xr;r1^x) +]t-ltrtrkx^x) takingthe gradient, Vf(xk+1;(2)

Vf(xk+1;=6, Setting

and Note that we have solvedfor a vector,i.e. Ax, which has both a steplength direction. Exampleof Newton'sMethod

- 2x,1x2 + +xlfromthepoint t=x?., *= [-i ] Wewishto optimize thefunction,

=[rl] At this pointVf(xo1 and the Hessian,. =[-3 -3] rhe Hessianinverseis |-0.6667 0.1666il grven o y : 1 0 . 1 6 6 60 7. 1 6 6 6 7 1I.n u s 0.66670.1666il[-81 =l-3 I =- | AX t oooz 0.1 66671 Lo. Lt+.1L-tI So,

xl ='"+Ax

[;t].[-t,]=[3]

3 2
{
t

0 H i (\> < r
- t

<

-z

-3
-4
I I

-6

-5

-4

-3

-2

-t

O 1 XI -AXIS

Ouasi-Newton Meth descent: and steepest with NeMon'smethod Letssummarize the problems

descentand gradually We want to developa methodthat startsout like steepest doesn't have trouble becomesNewton'smethod,doesn'tneed 2nd derivatives, exist. They Fortunately such methods diverge! doesn't functions and with eccentric methodwithoutthe descentand Newton's combinethe good aspectsof steepest drawbacks. by the expression In general we will defineour searchdirection g = -NVf(x) matrix." WhereN will be calledthe "direction lfN=l,thens= lf N =;{-1, thens = (3)

Unconstrained Optimization 7 lf N is alwayspositive thens: definite, you show Can this?

A RankOne Hessian Update Inverse with Taylor's Starting series: = f(x k; + Vf(xk;rsr **axkrH Axk f(xk+r1 wheretrakVf(xk+1;; taking Defining f=Vf(xk+t; -Vf(xr) f = grtrxr Thisshowsthat: or (H-t;xf =Axk (5) (6)
(4)

We will makethe direction matrixsatisfy this relationship. However, sincewe can calculatef and Axk on|y after the line search,the direction matrixNk does not usuallysatisfythis relationship, but we will make pk+l,yk =Axk How can we do this? is: One of the simplestpossibilities = Nk + Ek = Nk + auuT 1;k+1 (g) (7)

WhereEk is a "correction" matrix,whichwe will assumeis of the form auuT , which matrix. is a rankone symmetric (Nk+ auuT)into (7): lf we substitute NhA+ auurf = Axk or - Nhf) auurf = (A:rk (9) (10)

Un constrain ed Opti mization

Noting that uTf is a scalar Then u mustbe proportional to (Axk- Nkt') Sinceany changein lengthcan be absorbed by a, we will set u = (Axk-Nkt' ) Substituting this into(10): a(Axt<-Nt<^l )r I = (AxlLNk^yk ) ) (Axk-Nk'yk Forthisto be true,then

a (Axtcru\f ;r f = 1
so aI (lxk-N kTk )r Tk (11)

Thusthe direction matrixat any stageis givenby:

pk+1- pk.r-

(Axx-Xry<Xlxk-Nkf)r

(Axr-rurY<;r tt

(12)

property." The nextstep is to showthat this 'rank one update" has the "hereditary property will The hereditary statesthat not only (7),i.e. Pk+l s311sfy
pk+1yk

but

tr1k+1f-1 = p k+1.yk-2= p k+lp =

Axk-1

Axk I
|

Axk-2 I
Aro )

(13)

Why is this significant?Let'swritethis relationship as follows: pk+11,/<,^/r. . . fl - 1A:xk,Axk-1,6ak-2. . .Axol

Let the matrixdefinedby the columnsof y be denotedby G, and the matrixdefined by columnsof Ax be denotedby X. Then,

Unconstrained Optimization g 1;k+1 e =X lf t'...f are independent, andif wehavenvectors, i.e. G isasquarematrix, then the inverse for G existsand is unique and = X G-1 11k+1 is uniquely defined. (13),thenwe havethe important Sincethe hessian inverse alsosatisfies result that, after n steps the direction matrixbecomes the hessianinverse for a quadratic function. Numerical Inverse Example of RankOne Hessian Update

Wewishto minimize *l -Za*2 * 4*3 thefunction

startins from xo=f?]

=h?] vf(xo1

we let No= [; ? ." thesearch direction is ]

$ =

=l-?.?22] we execute a line search inthis direction and stop rt x1=[-3:9ti Vf(xr;

rhen

=x1-xo =[-3:9t] Axo f?] =[_?1r]

- t;l] = - v(,p) =l:.?n] f =vf(xl) [_?;133r]

- Nor=t-:,'t -t;?][-?;133r] =[;3:333] then axo
(Axo-NoToxAxo-Nof)r _

(^xo-Noro)r^/o

1s B22l r-4s66 [;3:333]

1s.822] [_4.s66 [_?;1BBr]

10

Unconstrained Optimization

-60.35 [ -1e.01 I 6 0 . 3 5 1 9 1 . 0J 5 L
-237.97 -.080 .254 = [ .254 -.803 f L I

pk+1 =[I ?].[ ;3?-""1 ]

.ZS+ 1 =rL .92 . z s. + tgz )
New searchdirection: .92 .2541 l-2.6641 $= -l| .zsq .rgz..| L-t.szz)

12.8371 =L .gzoJ = at theoptimum = we arrive Whenwe stepin thisdirection, [3] Vf(xe; [3] "

_'1=[B]_f3:9tJ=[r:9t] =x2 Axl

rol [_2.6641 12.6641 -l_1.sizl= = =yf2 vfl T1 LoJ Li.;;iJ

-l ::^ ?3+ll?8221 - N1yl) =[':9'] u=(trvl

= [':9']-ti'fl=[::!ll
?uur= r, 604l L -.og -.os I -.28] [-.81 li'.szzt

-28] -.26 r-.81 - oe [::311 r =|

r .s2 .2s4f . f -.26 -.oe] _ f .66 .16 I =1.:;A N2=Nl +auur _:mI = L .ro .167 ;e; J . L_:oe )

Unconstrained Optimization 11 for the Rank1 Update Property Proofof the Hereditary matrixand suppose symmetric thatAxO, Axl,..., Axk Theorem.Let H be a constant wherey' =HAxi,i=0,1,2,...,k. with any Starting and f, T1,...,t'are givenvectors, matrixNo,let initialsymmetric
pk+1- pk..

t<'yr) r (u k-Nkf) (axr- trt

(Axk-trtk',X;r tt

(14)

then pk+11 = Axi fori( k. (15)

point,k, the following We knowthat at the current Proof.The proofis by induction. is true, (16) Nk+1f= Axk the update. Now, becausewe enforcedthis conditionwhen we developed supposeit is truethat, N$ = 61i for i ( k-1 (17)

property matrix. (This holdsfor the previous direction i.e. that the hereditary We can post "suppose" hereis whatmakesthis a proofby induction.) assumption (1a)by t', giving, multiply pk+1f=Nkf +

kf) (lx t<-n r'yrlry (Axk-N

(Axr-rur1;r tt

(18)

Tosimp|ifythings,|etyk=ffisothatwecanwrite(18)as, 1k+1f= Nk/ + yrlUururfl;ry (19)

We can distributethe transposeon the last term, and distributethe post multiplication by t' to give,

pk+1f= Nkf + yrllorrlry- (f)rNryl

(17)is true,we can replace Nkt'withAxi: Sincewe haveassumed - (f)r^xil pk+1.,/= Axi + yt<11lxr;ry Nowwe examine the term in brackets.We notethat.

(20)

(21)

12

UnconstrainedOptimization

(f)rUi

= (HlxklrAxi = (Ur;rgOri = 161k;ry

(22)

giving, in (21)vanishes, So the termin brackets

pk+1f= 4yi

(23)

property holdsfor the previous direction matrix,it holdsfor Thus,if the hereditary (16)is all that is needed matrix. Whenk=0,condition the currentdirection to have property N1. The secondupdate, forthe first update, N2 will then the hereditary property havethe hereditary sinceN1 does,and so on. (BFGS)Update The Broyden-Fletcher-Goldfarb-Shanno property, it does not Although the aboverankone updatedoeshavethe hereditary positive guarantee matrix,N, is definite.Why is it that at each stagethe direction important havethis property? thatthe update

positive Davidon a ranktwo update that guarantees was one of the firstto propose by Fletcher and Powell; thus this definiteness. His methodwas furtherdeveloped (DFP)method. This methodcameto be knownas the Davidon-Fletcher-Powell updateis,

1;k+1= tt* *'

e*(S*lt - NqA(f)TN
(Axk)rTk

(f)rNqf

(24)

was popular of positive This update for manyyears. The property definiteness of N requires that a minimizing step,whichwe havedenoted as cr*,be takenin the line fit andthereis usually thatwe founda* usinga parabolic search. Recall someerror the parabolaseveraltimes as we obtain here. We can reducethe enor by refrtting this requires morefunction morepointsin the regionof cr*. However, evaluations. The DFP methodis quitesensitive to errorsin cr*and can degrademarkedlyif cr. is not accurate. by all four The current"bestuupdateis known as the BFGS update,suggested independently as the DFP updatebut is authors in 1970. lt has the sameproperites less sensitive to erors in cr". This meanswe can be "sloppy"in our line search when we are far away from the optimumand the methodstill works well. This updateis,

,, (C).I!{) pk+1-nr+ + ' \jffil[r *

(.'

Axt<(a:rr)r _ ur('/1rNr+nhA(o<r)r (25) "';: ' iii';t-' (ur1ry, (^xqrf J tarqrf

Unconstrained Optimization 13 Hessian Inverse Updates Vs. Hessian Updates so far are updates All of the updates we havepresented for the HessianInverse. for the Hessian itself,as will be required We can easilydevelop updates for the starting fromthe condition SQP algorithm, f=6r4xr (26)

insteadof (H-t;rTk = AxK which we used before. The BFGS Hessian (Eqn15 is the Hessian inverse approximation) is givenby, approximation

'=HK. 6k+1

'vkrr/<tr

gk4ak161k1Tg

(tfiirn C.k)fik^rk

e7)

You will note that this looksa lot like the DFP Hessianinverseupdatebut with H withAx. In factthesetwo formulas are said interchanged with N andl interchanged to be complementary to eachother. Questions methods? Why are thesemethods calledquasi-Newton How can we obtainthe Hessian,or Hessianinverse,if we never evaluate2nd derivatives? see R. Fletcher, PracticalMethodsof For more information on Hessianupdates, Wiley,1980. Optimization,

gEF TSF;
E'
c n .o B :i; .'.

-,-i ?*_'*$4-B#:.sn+t
'-',*i &-*ffi J'lj;:$ii'

:-f._ .

'

giii{i:Eii
f; t:-..

'' :ifl,*.Ple:'lil "*i

i : .:

"i

i -

--'\

- ' :')t' '.\

."

'

-:1:ri . -

---

::iE 5li;ii
F
:
*

; ! ; : 3 ri i i ai
i- -.'
j

3.

- - !
"\\"

c*=r-:
" D i ' s = ' p

iii ' iF iE
e

i ii'i;i; ni1 =
t q

ritirli ;i=
_

l i i i i g I ! l qi
i l . o '

N
-

'

>'.
-\.

r s

's
'

?-, ;-

i_ $ . - -

-\ \v

\. " \ {

c\ d

s
t

-*\

^F:q
\5
4' a

:
s

\ l
\

\) s

S.

,.

:t'-c r F P \-r
s '
}t-1 :v

r'\i > s^

-'-

s' \S A \ \.

{ --r\
-\
. l l
Y "l

>,

{,

cb

-f-\ sv F

r+

a . t\

.N
\i

\'

\-)
\-. \

\J

1{ J
i

t \,a
v

{ n\r\ $

--... - \,.. -. A
' a

\*

G'

E
!
l

* u
- 'er L\
1

rtnQ

- ; r:' ?f1"ffifff/f, M-,, : i iiiiiiiillil N 61..'r\*.,1,i. ''r'i*N:,\ /i1,,;:,;/;W'lll,' j 4f

i \-{'r\-'.'\\,q

i iitttitttttltt i n c<\t+r\ i \,.i\i\ii:.:fl'2'i7r'i''' l,/' r ililitttWr !\i\ i i 't j r\ i\. i lr)\\'r%t llillllllllllf! 3i-\ i ,'"1\,'$':,::-22//r,i/i lllll,jlllllllf I i
---/--.'/,//r't'iJ//!il / I Iiiiittfiitttti ! r

ll';llll,ljffi *-4fi:;,,1',lffill i r77n',rl/1,'

r 1 /

l \ \ \i t ' + - - '-;-;\ - ' /t/t / t/i/ -/',../ t'/ li\\it

'f

i \ . \ _ , o . / , , , , /// / i , r i l l l l i i l x i il # l l r a th\\\\\ r ,rj' lri/t r,r!Ufitt#ititiiit i I

*i

A-t-

l r l t t r t , r i ,itit t t | !titr!*ijtiti -; ;

! rN\\ -N\\i:\\\\\ lfi::ttli11///ffiff f l,l i A ,/' ,,- J i

) '

: ;
i

ri t {r,, iriffi\ I iY 1l#lllllllll li 8L\\{"i'N\ \hL Y/lllllllllllllll

+ \\\t\\\\l\\\\ \ 1{.-J, --rJ ,' ,!1t;l!i,!ttii\! i I

: .. : I :
l I ! i' I

-iiI

i,. -__ I

..

i r __ __i
. _Z

ri\i\.i'r'.\ \

-,-, :

--.-- ---':r..:.1:--:.-:-; ,.-. ir.-i- -a ji:Jj ., : ,li .,i..,.

j I=j1-,rr::-

'-r- -Z
,

:1

-'-1@'-----: -j ''
--:gqEE#l#'F-;:;r.;-.

*e#::----.*.-.-..--.--

Xf,=AXfS

tr,'5i'i-.-

J I
i,

i.

.,,--rz::lt, '.t..
/_..-_.4::

ffi.fu;g:i*-J2*.*ffi5"..'..,:'

?*Llr^^ 4 tsFcs $A n"ld Exo^^-ple . 'kJ '-' -1x, '

F y,r- Z>-Y,t f + X,t + F
t

'

A ' v r ' = i :-l--rl ^ L 6 J { 6 f l l l r J t_o J ;x'=LrJ j-t-:: LzK Li f "'u, I i*,= [-r,tozl 1,= r fl

rqt

a lt I

?
r

f-: lP
t u

J i I r.4rr",
i

'=3izL I L'

= 5l' + [. iry, lsw !

i

1tt5 'S*etl

,,rffijJ

, 1 w 1I1 ':t 3+t3I I I .s1#?-J L,1tlt1't

-.inlz? ,, f. v1("11 I

tsxol1 :, . l=,,

= l.z*usl .,X " L

'f-t'r88?t'.l | ="r1,cr c l" -zs'rt'r'J |ste: J l_ or -, Ba s(rll '':^':::.1 f , Egre t f * 'J' N' f'fi"'
-l.lllt7 z7z -6t l:, I - t.,,lr? -J-tonr | -J
I

'r-

'-i.t r,t

LltE?6J

l-.iose= t,6ttss1

s.

, f ,t*zr I
t

l-.oeltL I
fi:,;T,.E('tg

[*'ttzrrJ
:::__-_ -.

l:nwl
h;-;H-i-

.")
P

-l-,1afl: I a 7 -1.?1778
J
.

l '

J.

.jn,t1 l 't.oz-o1L),
bo zt.

.''
':, ,r..+
t!:jii& : :.1