Optimisation Problems

1/32

Dr. Anupap Meesomboon

Optimisation

1/32

We have a rope of length a to tie a box from top to bottom along the two perpendicular directions. What is the maximum volume that such a box can contain?

2/32

Dr. Anupap Meesomboon

Optimisation

2/32

Suppose that a certain electrical network consists of three dierent channels through which electric power ows. If xi , i = 1, 2, 3, stands for the amount of power through channel i, the total loss in the network is given by the function x2 1 2 3 ) p(x1 , x2 , x3 ) = x3 + (x2 1 + x2 + 2 10 If a total amount of r is to be transferred, determine the amounts through each channel so as to minimize the loss of power. Evidently, the problem reduces to nding the minimum of the above function providing a measure of the loss of power under the constraints x1 + x2 + x3 = r, xi > 0

3/32

Dr. Anupap Meesomboon

Optimisation

3/32

Suppose a wireless broadcast system has n transmitters. Transmitter j broadcasts at a power of pj 0. There are m locations where the broadcast is to be received. The path gain from transmitter j to location i is gi,j ; that is, the power of the signal transmitted from transmitter j received at location i is gi,j pj . The total received power at location i is the sum of the received powers from all the transmitters. We are interested in nding the minimum sum of the transmit powers subject to the requirement that the received power at each location is at least P .

4/32

Dr. Anupap Meesomboon

Optimisation

4/32

Consider a wireless communication system. There are n mobile users. For each i = i, . . . , n, user i transmits a signal to the base station with power pi and an attenuation factor of hi (i.e. the actual received signal power at the base station from user i is hi pi ). When the base station is receiving from user i, the total received power from all other users is considered interference (i.e. the interference for user i is j =i hj pj ). For the communication with user i to be raliable, the signal-to-interference ratio must exceed a threshold i . We are interested in minimising the total power transmitted by all the users subject to having reliable communications for all users.

5/32

Dr. Anupap Meesomboon

Optimisation

5/32

and many more . . .

6/32

Dr. Anupap Meesomboon

Optimisation

6/32

An optimisation problem is of the form minimise f (x) subject to x where f : Rn R, a real-valued function, is called the objective function or cost function or just cost,

7/32

Dr. Anupap Meesomboon

Optimisation

7/32

An optimisation problem is of the form minimise f (x) subject to x where f : Rn R, a real-valued function, is called the objective function or cost function or just cost, x = [x1 x2 xn ]T = [x1 , x2 , . . . , xn ]T = (x1 , x2 , . . . , xn ) Rn is an n-dimensional column vector and is called the minimiser or minimum point,

7/32

Dr. Anupap Meesomboon

Optimisation

7/32

An optimisation problem is of the form minimise f (x) subject to x where f : Rn R, a real-valued function, is called the objective function or cost function or just cost, x = [x1 x2 xn ]T = [x1 , x2 , . . . , xn ]T = (x1 , x2 , . . . , xn ) Rn is an n-dimensional column vector and is called the minimiser or minimum point, the variables x1 , x2 , . . . , xn are called decision variables,

7/32

Dr. Anupap Meesomboon

Optimisation

7/32

An optimisation problem is of the form minimise f (x) subject to x where f : Rn R, a real-valued function, is called the objective function or cost function or just cost, x = [x1 x2 xn ]T = [x1 , x2 , . . . , xn ]T = (x1 , x2 , . . . , xn ) Rn is an n-dimensional column vector and is called the minimiser or minimum point, the variables x1 , x2 , . . . , xn are called decision variables, is a subset of Rn and is called the constraint set or feasible set.

7/32

Dr. Anupap Meesomboon

Optimisation

7/32

Some optimisation problems are to maximise the objective function, i.e. maximise f (x) subject to x We can reformulate the above maximisation problem as minimise f (x) subject to x

8/32

Dr. Anupap Meesomboon

Optimisation

8/32

The problem minimise f (x) subject to x is called a constrained optimisation problem. If = Rn , the problem is said to be an unconstrained optimisation problem.

9/32

Dr. Anupap Meesomboon

Optimisation

9/32

The constraint x is called a set constraint.

10/32

Dr. Anupap Meesomboon

Optimisation

10/32

The constraint x is called a set constraint. The constraint of the form = {x : h(x) = 0, g (x) = 0}, is called a functional constraint.

10/32

Dr. Anupap Meesomboon

Optimisation

10/32

Denition
(Local minimiser) Suppose that f : Rn R is a real-valued function dened on some set Rn . A point x is a local minimiser of f over if there exists > 0 such that f (x) f (x ) for all x \ {x } and ||x x || < .

11/32

Dr. Anupap Meesomboon

Optimisation

11/32

Denition
(Local minimiser) Suppose that f : Rn R is a real-valued function dened on some set Rn . A point x is a local minimiser of f over if there exists > 0 such that f (x) f (x ) for all x \ {x } and ||x x || < . (Global minimiser) A point x is a global minimiser of f over if f (x) f (x ) for all x \ {x }

11/32

Dr. Anupap Meesomboon

Optimisation

11/32

If we replace with > in the above denition, then we have a strict local minimiser and a strict global minimiser, respectively.

12/32

Dr. Anupap Meesomboon

Optimisation

12/32

Let f : Rn R. The rst-order derivative of f , denoted by Df , is f f f Df = , ,..., x1 x2 xn The gradient f = (Df )T .

13/32

Dr. Anupap Meesomboon

Optimisation

13/32

The second-order derivative of f , called the Hessian of f , is 2f 2f (x) x (x) x x2 n 1 1. . . . H (x) = F (x) = D2 f (x) = . . 2f 2f x1 xn x2
n

14/32

Dr. Anupap Meesomboon

Optimisation

14/32

Denition (Feasible direction)

A vector d Rn , d = 0, is a feasible direction at x if there exists 0 > 0 such that x + d for all [0, 0 ].

15/32

Dr. Anupap Meesomboon

Optimisation

15/32

Theorem (Chain Rule)

Let g : D R be dierentiable on an open set D Rn , and let f : (a, b) D be dierentiable on (a, b). Then, the composite function h : (a, b) R given by h(t) = g (f (t)) is dierentiable on (a, b), and f1 (t) . h (t) = Dg (f (t))Df (t) = g (f (t))T . . fn (t)

16/32

Dr. Anupap Meesomboon

Optimisation

16/32

Denition (Directional Derivative)

Let f : Rn R and d be a feasible direction at x . The directional derivative of f n the direction d, denoted f , is d f f (x + d) f (x) = lim d 0

17/32

Dr. Anupap Meesomboon

Optimisation

17/32

If ||d|| = 1, then f is the rate of increase of f at x in the d direction d. To compute f , d f (x) = d df (x + d) d

T

=0

= (f (x)) d = dT f (x) = f (x), d

18/32

Dr. Anupap Meesomboon

Optimisation

18/32

Denition (O, o Notation)

Let g be a real-valed function dened in some neighbourhood of 0 Rn , with g (x) = 0 if x = 0. Let f : Rm be dened in a domain Rn that includes 0. Then, we write 1. f (x) = O(g (x)) to mean that there exists a number K > 0 and > 0 such that if ||x|| < , x , then ||f (x)|| K |g (x)|

19/32

Dr. Anupap Meesomboon

Optimisation

19/32

Denition (O, o Notation)

Let g be a real-valed function dened in some neighbourhood of 0 Rn , with g (x) = 0 if x = 0. Let f : Rm be dened in a domain Rn that includes 0. Then, we write 1. f (x) = O(g (x)) to mean that there exists a number K > 0 and > 0 such that if ||x|| < , x , then ||f (x)|| K |g (x)| 2. f (x) = o(g (x)) to mean that ||f (x)|| lim =0 x0,x |g (x)|

19/32

Dr. Anupap Meesomboon

Optimisation

19/32

Theorem (First-Order Necessary Condition:FONC)

Let be a subset of Rn and f be a continuously dierentiable function (f C 1 ) on . If x is a local minimiser of f over , then for any feasible direction at x , dT f (x ) 0

20/32

Dr. Anupap Meesomboon

Optimisation

20/32

Denition
An -ball or -neighbourhood of a point x0 is B (x0 ) = {x | d(x, x0 ) = ||x, x0 || < }

21/32

Dr. Anupap Meesomboon

Optimisation

21/32

Denition
An -ball or -neighbourhood of a point x0 S is B (x0 ) = {x S | d(x, x0 ) = ||x, x0 || < } is called the radius and x0 is called the centre of the -ball. Some authors use the word neighbourhood to mean a set that contains an -ball.

22/32

Dr. Anupap Meesomboon

Optimisation

22/32

Denition
A point x S is an interior point of S if B (x) S for some > 0.

23/32

Dr. Anupap Meesomboon

Optimisation

23/32

Denition
A point x S is an interior point of S if B (x) S for some > 0. The set of all interior points of S is called the interior of S .

23/32

Dr. Anupap Meesomboon

Optimisation

23/32

Denition
A point x S is an interior point of S if B (x) S for some > 0. The set of all interior points of S is called the interior of S . A point x S is a boundary point of S if every -ball of x contains a point in S and a point not in S .

23/32

Dr. Anupap Meesomboon

Optimisation

23/32

Corollary (FONC:Interior case)

Let be a subset of Rn and f C 1 a real-valued function on . If x is a local minimiser of f over and x is an interior point of , then f (x ) = 0

24/32

Dr. Anupap Meesomboon

Optimisation

24/32

Suppose f C 2 . Let x and x0 be points in Rn , and let z () = x0 + (x x0 )/||x x0 || Dene : R R by

Taylor series expansion of a real-valued function f : Rn R about the point x0 Rn

() = f (z ()) = f (x0 + (x x0 )/||x x0 ||)

25/32

Dr. Anupap Meesomboon

Optimisation

25/32

Applying the chain rule gives

26/32

Dr. Anupap Meesomboon

Optimisation

26/32

Quadratic Forms
Denition
A quadratic form xT Qx is called positive denite if xT Qx > 0 for x = 0 (we write Q > 0).

27/32

Dr. Anupap Meesomboon

Optimisation

27/32

Quadratic Forms
Denition
A quadratic form xT Qx is called positive denite if xT Qx > 0 for x = 0 (we write Q > 0). positive semidenite if xT Qx 0 for all x (we write Q 0).

27/32

Dr. Anupap Meesomboon

Optimisation

27/32

Quadratic Forms
Denition
A quadratic form xT Qx is called positive denite if xT Qx > 0 for x = 0 (we write Q > 0). positive semidenite if xT Qx 0 for all x (we write Q 0). negative denite if xT Qx < 0 for x = 0 (we write Q < 0).

27/32

Dr. Anupap Meesomboon

Optimisation

27/32

Quadratic Forms
Denition
A quadratic form xT Qx is called positive denite if xT Qx > 0 for x = 0 (we write Q > 0). positive semidenite if xT Qx 0 for all x (we write Q 0). negative denite if xT Qx < 0 for x = 0 (we write Q < 0). positive semidenite if xT Qx 0 for all x (we write Q 0).

27/32

Dr. Anupap Meesomboon

Optimisation

27/32

Theorem (Second-Order Necessary Condition:SONC)

Let Rn , f C 2 a function on , x a local minimiser of f over , and d a feasible direction at x . If dT f (x ) = 0, then dT F (x )d 0 where F is the Hessian of f (i.e. F (x ) = D2 f (x )).

28/32

Dr. Anupap Meesomboon

Optimisation

28/32

Corollary (SONC:Interior case)

Let x be an interior point of Rn . If x is a local minimiser of f : R, f C 2 , then f (x ) = 0 and F (x ) is positive semidenite (F (x ) 0); that is, for all d Rn , dT F (x )d 0

29/32

Dr. Anupap Meesomboon

Optimisation

29/32

Theorem (Second-Order Sucient Condition:SOSC,Interior Case)

Let f C 2 be dened on a region in which x is an interior point. Suppose that 1. f (x ) = 0 and 2. F (x ) > 0 (F (x ) is positive denite) Then, x is a strict local minimiser of f .

30/32

Dr. Anupap Meesomboon

Optimisation

30/32

31/32

Dr. Anupap Meesomboon

Optimisation

31/32

32/32

Dr. Anupap Meesomboon

Optimisation

32/32