Division of Scientific Computing

Department of Information Technology

Uppsala University

Optimization
Written examination 2012-08-20 — Solution

1. Consider the linear program

max 2x1 + 3x2

subject to 2x1 + 7x2 ≤ 6
(1)
3x1 + 4x2 ≤ 12
x1 , x2 ≥ 0

Solve the problem with the simplex method with Bland’s rule (without using the
tableau). Start with the slack variables as basic variables. Give both the maximum
and the maximizer. [6pt]
2. Let us again consider the linear program (1).
(a) Write down the dual of problem (1). [1pt]
(b) State the complementary slackness conditions. [1pt]
(c) Solve the dual using complementary slackness if the primal maximizer is x∗ =
(3, 0)T . [2pt]
3. Consider a linear program
min cT x
subject to Ax ≤ b (2)
x≥0

with b < 0. Explain why you cannot use slack variables as initial basic feasible solution. [2pt]
4. (a) Write down the damped Newton algorithm (with modified Hessian). [2pt]
(b) Where does damped Newton differ from basic Newton? Explain why these
changes improve the algorithm. [2pt]
(c) Explain the term convergence rate. What is the rate for basic Newton? [2pt]
5. The internal resistance of a battery is a simple model for the complex electrochemical
processes involved during its usage. The internal resistance B of a battery can be
computed by comparing the nominal battery voltage N with the actual voltage V
that is measured when a device of resistance R is applied to the battery through the
formula
B
V =N .
B+R
For a battery of unknown internal resistance B and nominal voltage N , you are given
measurements of V for several different resistances R according to the following table.

R V
0.30 2.5
1.0 1.8
3.0 0.90
8.0 0.40

Find the parameters that fit the data best in nonlinear least squares sense. Formulate
the optimization problem and perform one step with the Gauss–Newton algorithm for
the initial guess N (0) = 5 and B (0) = 1. [6pt]

1
6. Consider the optimization problem

min f (x1 , x2 ) = sin(x1 ) cos(x2 )

π 3
subjecto to ≤ x1 ≤ π (3)
4 4
5
π ≤ x2 ≤ π
4
(a) Write down the Karush–Kuhn–Tucker conditions for (3). [2pt]
(b) Check whether or not (x1 , x2 ) = ( π2 , π) is a KKT point. [2pt]
(c) Is the optimization problem convex? [3pt]

The trigonometric identities sin2 (x) = 1

2 (1 − cos(2x)) and cos2 (x) = 1
2 (1 + cos(2x))
might help you to solve this problem.
7. Consider the optimization problem

min f (x1 , x2 ) = x21 + x22

(4)
subjecto to x1 + x2 = 4.

(a) Determine the solution analytically. [1pt]

(b) Rewrite (4) so that the quadratic penalty method can be used to solve the prob-
lem. [1pt]
(c) Determine how the solution with the penalty method depends on the large pa-
rameter ρ. Also determine the optimal solution as ρ → ∞. [3pt]

Good luck!
Katharina

2
1. Rewrite the problem to standard form
min −2x1 − 3x2
subject to 2x1 + 7x2 + x3 = 6
(5)
3x1 + 4x2 + x4 = 12
x1 , x2 , x3 , x4 ≥ 0.
• basic indices: x3 , x4 , nonbasic indices x1 , x2
•    
1 0 2 7
B= , N= (6)
0 1 3 4
xB = B −1 b = (6, 12)T , cB = (0, 0)T , cN = (−2, −3)T
• simplex multiplier: y = (0, 0)T , reduced costs: ĉN = (−2, −3)T
• both entries in ĉN negative, take the one with smallest index by Bland’s rule:
iin = 1
• Âiin = B −1 Aiin = (2, 3)T
• min{ 26 , 12
3 } is attained for index iout = 3
• basic indices: x1 , x4 , nonbasic indices x3 , x2
•    
2 0 1 7
B= , N= (7)
3 1 0 4
xB = B −1 b = (3, 3)T , cB = (−2, 0)T , cN = (0, −3)T
• simplex multiplier: y = (−1, 0)T , reduced costs: ĉN = (1, 4)T
• The reduced costs are positive, i.e. we have found the optimal solution.
• Solution to the original problem: (x∗1 , x∗2 ) = (3, 0) and the value of the objective
function is 6.
2. (a) The problem is in canonical maximization form. The dual, hence, is given by

min 6y1 + 12y2

subject to 2y1 + 3y2 ≥ 2
(8)
7y1 + 4y2 ≥ 3
y1 , y2 ≥ 0

(b) The complementary slackness conditions are: xT (c − AT y) = 0 and y T (b − Ax) =

0, where      
2 7 6 2
A= , b= , c= .
3 4 12 3
(c) For the solution of the primal computed in part (a), it holds that (b − Ax∗ ) =
(0, 3)T . Since the second constraint is not satisfied, y2∗ = 0. Since x∗1 is non-zero,
the first constraint of the dual must be satisfied: 2y1∗ + 3y2∗ = 2, i.e., y1∗ = 1. As
in the primal, the optimum is 6.
3. To solve the problem with the simplex algorithm, we transform it to standard form

min cT x
subject to −Ax − s = −b
(9)
x≥0
s≥0

Choosing s as basic variables and x as non-basic, the constraint −Ax − s = −b gives

s = b. Then, s < 0 and hence the positivity constraint s ≥ 0 is not satisfied which is
why the slack variables do not give a feasible solution.

3
4. (a) The damped Newton algorithm reads
i. Choose initial guess x0 , tolerance ε, and parameters µ, σ.
ii. For k = 0, 1, 2, . . .
A. If k∇f (xk )k ≤ ε, stop.
B. Compute modified Cholesky factorization√ LLT of ∇2 f (xk ) where the
diagonal elements in L are at least σ.
C. Solve LLT pk = −∇f (xk )
D. Line search:
αk = 1
while f (xk + αk pk ) > f (xk ) + µαk ∇f (xk )T pk
αk
αk =
2
end

E. Set xk+1 = xk + αk pk .
(b) Compared to Newton, one uses the modified Cholesky decomposition. The modi-
fication replaces the Hessian by a (well-conditioned) positive definite matrix (with
eigenvalues larger or equal µ) which makes sure that the search direction is a suf-
ficient descent direction. Moreover, we use the Armijo rule with backtracking
as a line search strategy to make sure we take a step for which the new itera-
tor is an improvement to the old one and the step size is not too small. These
modifications make sure the method is globally convergent.
(c) The sequence {xk }k∈N ⊂ Rn converges to x∗ with rate p and constant C, if
kxk+1 − x∗ k
lim = C. (10)
k→∞ kxk − x∗ kp
The convergence rate of basic Newton is two if one is close enough to the mini-
mizer.
5. We define the residuals
B
fi = N − Vi , i = 1, 2, 3, 4,
Ri + B
where Ri and Vi denote the measured data. We define x = (N, B). Then, the residuals
are functions of x. With these residuals, the best fit is given by the minimization
problem
4
1X
min fi (x)2 .
x∈R2 2
i=1

We solve this problem with the Gauss–Newton method with initial guess x(0) = (5, 1)T .
• We get the initial residual vector
F (0) = (1.35, 0.7, 0.35, 0.156)T .
We compute the gradient of the residual vector with respect to the vector x,
B
 ∂fi  !  
(0) ∂N Ri +B 0.77 0.5 0.25 0.11
∇F = ∂fi =   = .
∂B i=1:4
N Ri1+B − (Ri +B)
B
2 0.89 1.2 0.94 0.49
i=1:4

The Gauss–Newton method computes the update p(1) as

   
0.92 1.6 (1) −1.5
   
∇F (0) (∇F (0) )T p(1) = − ∇F (0) F (0) ⇔ p =
1.6 3.5 −2.5
for which we get p(1) = (−1.93, 0.177)T and x(1) = x(0) + p(1) = (3.07, 1.18)T .
This means that the voltage is approximately N = 3.07 and the battery’s internal
resistance is B = 1.18.

4
6. (a) The Lagrange function for this problem reads
 
 π 3
L(x, λ) = sin(x1 ) cos(x2 ) − λ1 x1 − − λ2 π − x1
4 4
 
5
− λ3 (x2 − π) − λ4 π − x2 .
4
This gives the KKT system
   
cos(x1 ) cos(x2 ) − λ1 + λ2 0
∇x L = =
− sin(x1 ) sin(x2 ) − λ3 + λ4 0
π 3π 5
≤ x1 ≤ , π ≤ x2 ≤ π
4 4 4 (11)
λ1 , λ2 , λ3 , λ4 ≥ 0
   
 π 3π 5
λ1 x1 − + λ2 − x1 + λ3 (x2 − π) + λ4 π − x2 = 0.
4 4 4

(b) The primal constraints are satisfied. The first, second, and the fourth constraint
are non-binding and hence λ1 , λ2 , λ4 need to be zero to satisfy complementary
slackness. Then, the gradient conditions read
   
0 0
∇x L = = , (12)
−λ3 0

which means λ3 = 0. Hence, the KKT conditions are satisfied for x = ( π2 , π)T ,
λ = 0, so that we have a KKT point.
(c) An optimization problem is convex if (i) the feasible set is convex and (ii) the
objective function is convex on the feasible set. The feasible set is given by linear
inequality constraints and hence it is convex. The Hessian of the function f is
given as
 
2 − sin(x1 ) cos(x2 ) − cos(x1 ) sin(x2 )
∇ f (x1 , x2 ) = (13)
− cos(x1 ) sin(x2 ) − sin(x1 ) cos(x2 )
To show that f is a convex function on the feasible set, the Hessian needs to be
positive semidefinite on the feasible set. This is the case if − sin(x1 ) cos(x2 ) ≥ 0
and the determinant sin2 (x1 ) cos2 (x2 ) − cos2 (x1 ) sin2 (x2 ) ≥ 0. On the interval
[ π4 , 3π
4 ] the sine is non-negative and the cosine is non-positive on the interval
[π, 5π 4 ]. Hence, the element − sin(x1 ) cos(x2 ) is non-negative. The determinant
can be rewritten as 21 (cos(2x2 ) − cos(2x1 )) which is non-negative on the feasible
set (first term non-negative, second term non-positive). Hence, the optimization
problem is convex.
7. (a) First resolve the constraint for x2 : x2 = 4 − x1 . Inserting this into the objective
function gives the one dimensional function: g(x1 ) = x21 + (4 − x1 )2 . The first
derivative is g ′ (x1 ) = 2(x1 − 4 + x1 ) which is zero for x1 = 2. Since the second
derivative g ′′ (x1 ) = 4 is always positive, x1 = 2 is the global minimizer of g. This
corresponds to the minimizer (x1 , x2 ) = (2, 2) of (4), the minimum is 22 + 22 = 8.
(b) To solve this problem with the quadratic penalty method, we transform the con-
strained problem to the following unconstrained optimization problem
ρ 2
min ψρ (x1 , x2 ) = x21 + x22 + (x1 + x2 − 4) , (14)
2
where ρ > 0 is a large parameter.
(c) Problem (14) is an unconstrained optimization problem, so we solve it using the
first order optimality condition and set the gradient with respect to x to zero:
 
2x1 + ρ(x1 + x2 − 4)
∇ψρ (x1 , x2 ) = . (15)
2x2 + ρ(x1 + x2 − 4)

5
Subtracting the first and the second line of the gradient gives x1 = x2 . Inserting
this into the first equation gives 0 = 2x1 + ρ(2x1− 4) = (2 − 2ρ)x1 − 4. Hence,
2ρ 2ρ
the solution to the penalty problem is 1+ρ , 1+ρ . The Hessian is
 
2+ρ ρ
∇2 ψρ (x1 , x2 ) = ,
ρ 2+ρ

whose determinant is 2 + 2ρ > 0. Together with 2 + ρ > 0, this gives positive

definiteness of the penalty solution. In the limit ρ → ∞ the penalty solution
converges to (2, 2) which is the solution of the constrained problem found in part
(a).