Homework 1

Ana Huaman March 9, 2011

1. In class it was claimed that given a function h : h T u points in the direction in which h grows the most. Show that this is indeed the case Solution By denition, the directional derivative v h represents the instantaneous rate of change of the function h, moving in the direction of v . If h is dierentiable in u, we have: v h = h v h(u + v ) h(u) h T = v 0 u lim For property of escalar multiplication: h(u + v ) h(u) h T = 0 u lim where is the angle between v cos (1)
m

, its derivative

h T and v . u To nd the maximum rate of change of h moving in u direction (that is, to nd the biggest h(u + v ) h(u)), we need the right side of the Eq.1 to be the maximum. For cos , the maximum value it can take is 1, which happens when = 0 = 0 implies that the angle between that h T and v is 0, or in other words, u

h T points towards the same direction as v (they are parallel). As u h T we dened that the rate of growth is in the direction of v , and that u h T is parallel to it, in consequence also points to the maximum growth u direction.

2. Solve the following minimization problem: minu subject to the constraint that Au = c where Q = QT rank (A) = p Solution We know that the following equation must be satised to nd a minimum: h L + =0 u u First, we calculate L : u L= 1 T u Qu bT u 2 0 Moreover, A is a p m matrix (p m), with
m

1 ( uT Qu bT u) 2

L L(u + v ) L(u) = lim 0 u v Hallamos L(u + v ) L(u) rst: L(u + v ) L(u) = L(u+ v )L(u) = 1 1 (u + v )T Q(u + v ) bT (u + v ) uT Qu + bT u 2 2

1 T 1 1 1 1 u Qu+ uT Q v + v T Qu+ v T Q v bT ubT v uT Qu+bT u 2 2 2 2 2 1 T 1 T 1 T L(u + v ) L(u) = u Q v + v Qu + v Q v bT v 2 2 2 =

2

We cancel the 3rd term of the right side, because it has a factor which goes to zero. Grouping properly: 1 1 L(u + v ) L(u) = ( uT Q + uT QT bT )v 2 2 Replacing in the original equation: 1 1 ( uT Q + uT QT bT )v L(u + v ) L(u) 2 2 = v v L L(u + v ) L(u) 1 1 = = ( uT Q + uT QT bT ) u v 2 2 As stated in the initial conditions: Q = QT so we nally have: L 1 1 = ( uT Q + uT Q bT ) = (uT Q bT ) u 2 2

(2)

Second, to nd

h , we analyze h: u h : Au C = 0

Multiplying to both sides by C T h : C T Au C T C = 0 So we have: h = CT A u

(3)

Third, using 2 and 3 into the Lagrangian: L h + =0 u u uT Q bT + C T A = 0 We nd the value of u () from above: u = Q1 (b AT C ) Now we plug this u into h, to nd : h : Au = c Replacing 4 into the equation shown: AQ1 (b AT C ) = C Operating, we nd : = C T (AAT )1 AQ(AT A)1 AT (AQ1 b C ) CT C (5) (4)

So, our minimizer u would be: u = Q1 (b AT C ), where is the value obtained in Equation 5 2L Note: To make sure that u is a minimizer, we can analize , which u2 is Q, that is positive denite from the premise of the problem. Hence, the value u is safely considered a minimum. 3. Let F (M ) be the matrix function F (M ) = M T M M T where M is an n m matrix. What is the directional derivative of F? Solution The directional derivative is dened by: f (x, y ) = lim 3 f (x + y ) f (x)

For the function F in this problem, we nd F (M + N ) F (M + N ) = (M + N )T (M + N )(M + N )T F (M + N ) = M T M M T + M T N M T + N T M M T + 2 N T N M T + M T M N T + 2 M T N N T + 2 N T M N T + 3 N T N N T We nd F (M + N ) F (M ). We do also eliminate the factors which have n F (M + N ) F (M ) = M T N M T + N T M M T + M T M N T F (M + N ) F (M ) = MT NMT + NT MMT + MT MNT F (M, N ) = M T N M T + N T M M T + M T M N T The directional derivative of F is Equation 6 If f is a continuously dierentiable function, f : disprove that the directional derivative is given by f (x; y ) = f (x) y x
m

(6)

. Show or

Solution The directional derivative is dened by: f (x, y ) = lim f (x + hy ) f (x) h0 h (7)

As f is C 1 , we nd the Taylor expansion (h 0): f (x + hy ) = f (x) + Getting rid of the term O(h2 y 2 ) f (x + hy ) f (x) = f (x) hy x (8) f (x) hy + ((hy )2 ) x

f (x + hy ) f (x) f (x) = y h x Replacing the left side of 8 with 7: f (x, y ) = f (x) y x

so, the claim of this problem is true, if the initial conditions are met (function continuous and dierentiable). 4. Consider the volume maximization problem: Construct a box (closed on all sides) of maximal volume with sides x,y ,z , given an upper bound c on the area of the boundary of the box Solution 4

Dening formally our problem for maximizing the volume of a cube with sides x,y ,z : max V (x, y, z ) = max(xyz )
x,y,z x,y,z

We can re-dene the maximum like the opposite of the minimum of the negative of L, that is: max V (x, y, z ) = min L(x, y, z ) = min (xyz )
x,y,z x,y,z x,y,z

such that the area of the boundary is less than c: g (x, y, z ) : xy + xz + yz < c xy + xz + yz c < 0 Applying the Kuhn-Tucker conditions to nd a possible maximum (- minimum) for this situation: g L + =0 u u (where > 0). We know calculate each factor in the KT condition: L = (yz, xz, xy ) u g = (y + z, x + z, x + y ) u Applying both 9 in we get the following equations: yz + 2y + 2z = 0 xz + 2x + 2z = 0 xy + 2x + 2y = 0 From these equations we can nd x,y ,z in function of : x = y = z = 4 Applying this in the inequality condition g : g (x, y, z ) : 2xy + 2xz + 2yz c 962 c c 4 6 (9)

c We dene = ,so for this we would have a possible minimum of: 4 6 c x = y = z = 6 Checking the last condition: g (x , y , z ) = 0 5

c c c (2 + 2 2 c) = 0 6 6 6 (c c) = 0 we see that it is true. Hence, this is a probable minimum for L, which is the negative of the volume. Hence the maximum for the original problem is: c x = y = z = 6 5. Given an unconstrained optimization problem minu L(u) the normal FONC for optimality is L =0 u But what if we only have directional derivatives instead of normal derivatives. What is FONC in this case? Solution In some problems, it happens that we do not have normal derivatives. This due to the fact that not all directions of movement are feasible, meaning that if going in that direction, we are going to get out of the area where our problem is dened. An example of this is for the points that are located in the boundaries. Any direction that points outside the area is infeasible, whereas any direction pointing inside is feasible. For problems where we have only directional derivatives, we have to evaluate them like: f f (x + d) f (x) (x) = lim 0 d where d is a feasible direction and > 0 Back to the problem: We plan to calculate the Taylor expansion of: L(u + d) where we consider that u is a minimum with respect to cost L. To make things simpler, we dene the function: u() = u + d Note that u(0) = u . Let dene another function: () = L(u()) Note that (0) = L(u(0)) = L(u ), or the minimum cost we are considering. 6

Applying Taylor expansion to nd L(u + d) = (): () = (0) + (0) + O ( )

We can ignore the last element O(): () = (0) + dT L(u(0)) () = (0) + dT L(u ) Putting everything back to L and u : L(u + d) = L(u ) + dT L(u ) (10)

Now, if L(u ) is the minimum cost, then L(u + d) must be bigger than L(u ): L(u + d) L(u ) Replacing the left side with Eq.10 L(u ) + dT L(u ) > L(u ) dT L(u ) > 0 We are considering that > 0 (from the denition of directional derivatives). So, we can eliminate it safely: dT L(u ) > 0 (11)

is a condition for L(u ) to be a minimum, considering directional derivatives. So, the FONC for this case is given by Eq. 11, namely: dT L(u ) 0 6. Let L C 1 be convex, i.e., L(u1 + (1 )u2 ) L(u1 ) + (1 )L(u2 ), u1 , u2 Assume that L (u ) = 0 u Show that u is a global minimum to L
m

, [0, 1]

Solution Of the denition for a convex function: L(u1 + (1 )u2 ) L(u1 ) + (1 )L(u2 ), u1 , u2 We reorder the equation: L(u1 + (1 )u2 ) (L(u1 ) L(u2 )) + L(u2 ) L(u2 + (u1 u2 )) (L(u1 ) L(u2 )) + L(u2 ) 7
m

, [0, 1]

We substract L(u2 ) from both sides: L(u2 + (u1 u2 )) L(u2 ) (L(u1 ) L(u2 )) L(u2 + (u1 u2 )) L(u2 ) (L(u1 ) L(u2 )) We multiply and divide the left side of the inequality by (u1 u2 ) (u1 u2 ) L(u2 + (u1 u2 )) L(u2 ) (L(u1 ) L(u2 )) (u1 u2 )

If we consider 0, we would have that the second factor in the left side in the form of a derivative: (u1 u2 ) L(u2 ) L(u1 ) L(u2 ) u (12)

Now, from the initial armation, we have that a u such that: L (u ) = 0 u Let u2 = u , then replacing it in Eq. 12 (u1 u ) L(u ) L(u1 ) L(u ) u

(u1 u ) 0 L(u1 ) L(u ) 0 L(u1 ) L(u ) L(u ) L(u1 ) We change u1 by u and we nally have: L(u ) L(u) (13)

From the expression obtained in Eq.13, we easily see that L(u) is always greater or equal than L(u ); hence, it is a global minimum over the dominium of L ( m ), if L is a convex function in C 1